2021-07-26T05:54:12.715 INFO:root:teuthology version: 1.1.0-b6f3efab 2021-07-26T05:54:12.716 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292739 --verbose --name kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} --owner scheduled_kchai@teuthology -- /home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292739/orig.config.yaml 2021-07-26T05:54:12.742 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-07-26T05:54:12.785 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292739 branch: wip-kefu-testing-2021-07-26-1205 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '6292739' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-kefu-testing-2021-07-26-1205 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d workunit: branch: wip-kefu-testing-2021-07-26-1205 sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-07-26-1205 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_197e564fdaee12596de1dc8f3253d1704ae66d5d/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d targets: smithi094.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBDBcEEBoHh+nEOv9hbzC3zxBWQffei+99PooOzRMVELHWnzzyrZ6k9C8mTmAcBooWpKWNSl4eqd6QbaA31H/pxxnfiIMK0i19vLJd0hzODntQ7qXqUpfEiFeUW5pcjE26m8iBtXknNuUmgbNYFyC8EFPfV52Z6CzIvB+WqzhKx3P1m7sGPvKgI+lDE9wITNQWa6EKx8Al1mOPXpvAsr2oPUwtIB2SfIMCbMctCcHrMTV6g74fez8fKwf9YDBlITsve3DEfBadoF87MSvx8tRuWOhC0HR5W79zolVepH/122KWrJtRkbIrIxdwIWXHXkltwal/HDv1mIZOm2kmA4KYCgaMRJ97UGllOtCMeNMxHrP8jya3wAvm0CaZUQt3RDeBL4udr1K83wu6M3W2ylkGidOeeOeZ+oPuax9d0Gy0g7eIcQlHr3r2aUBLs6N88G1nvxAq8mMWjrGjBK/Tx4iTDBksEPCQfFEfQe1Nlhq0jGs2rugUSpRqmlQL7j2/LWU= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master teuthology_sha1: b6f3efab84b70e18e86ec271a07564aeb4ed3bac timestamp: 2021-07-26_05:46:54 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-07-26T05:54:12.785 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_197e564fdaee12596de1dc8f3253d1704ae66d5d/qa; will attempt to use it 2021-07-26T05:54:12.786 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_197e564fdaee12596de1dc8f3253d1704ae66d5d/qa/tasks 2021-07-26T05:54:12.786 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-07-26T05:54:12.788 INFO:teuthology.task.internal:Checking packages... 2021-07-26T05:54:12.808 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '197e564fdaee12596de1dc8f3253d1704ae66d5d' 2021-07-26T05:54:12.808 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-07-26T05:54:12.809 INFO:teuthology.packaging:ref: None 2021-07-26T05:54:12.809 INFO:teuthology.packaging:tag: None 2021-07-26T05:54:12.809 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-07-26-1205 2021-07-26T05:54:12.810 INFO:teuthology.packaging:sha1: 197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T05:54:12.810 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-kefu-testing-2021-07-26-1205 2021-07-26T05:54:12.971 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-6358-g197e564f-1focal 2021-07-26T05:54:12.971 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-07-26T05:54:12.982 INFO:teuthology.task.internal:no buildpackages task found 2021-07-26T05:54:12.982 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-07-26T05:54:13.016 INFO:teuthology.task.internal:Saving configuration 2021-07-26T05:54:13.026 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-07-26T05:54:13.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-26T05:54:13.059 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-07-26 05:48:53.571667', 'description': '/home/teuthworker/archive/kchai-2021-07-26_05:46:54-rados-wip-kefu-testing-2021-07-26-1205-distro-basic-smithi/6292739', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHS2yA6svsrRHdQtRq3HrkNp163rhTFPY21jY0OH4t5k', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi094.front.sepia.ceph.com'} 2021-07-26T05:54:13.059 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-07-26T05:54:13.070 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-07-26T05:54:13.070 INFO:teuthology.run_tasks:Running task console_log... 2021-07-26T05:54:13.122 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1057729d08>, signals=[15]) 2021-07-26T05:54:13.123 INFO:teuthology.run_tasks:Running task internal.connect... 2021-07-26T05:54:13.134 INFO:teuthology.task.internal:Opening connections... 2021-07-26T05:54:13.134 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2021-07-26T05:54:13.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:54:13.189 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-07-26T05:54:13.202 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2021-07-26T05:54:13.309 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2021-07-26T05:54:13.309 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2021-07-26T05:54:13.315 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="Ubuntu" 2021-07-26T05:54:13.315 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-07-26T05:54:13.315 INFO:teuthology.orchestra.run.smithi094.stdout:ID=ubuntu 2021-07-26T05:54:13.315 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE=debian 2021-07-26T05:54:13.316 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-07-26T05:54:13.316 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="20.04" 2021-07-26T05:54:13.316 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://www.ubuntu.com/" 2021-07-26T05:54:13.316 INFO:teuthology.orchestra.run.smithi094.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-07-26T05:54:13.317 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-07-26T05:54:13.317 INFO:teuthology.orchestra.run.smithi094.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-07-26T05:54:13.317 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_CODENAME=focal 2021-07-26T05:54:13.317 INFO:teuthology.orchestra.run.smithi094.stdout:UBUNTU_CODENAME=focal 2021-07-26T05:54:13.318 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2021-07-26T05:54:13.335 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-07-26T05:54:13.347 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-07-26T05:54:13.366 INFO:teuthology.task.internal:Checking for old test directory... 2021-07-26T05:54:13.367 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2021-07-26T05:54:13.373 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-07-26T05:54:13.390 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-07-26T05:54:13.391 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2021-07-26T05:54:13.422 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-07-26T05:54:13.450 INFO:teuthology.run_tasks:Running task kernel... 2021-07-26T05:54:13.467 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-07-26T05:54:13.467 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-07-26T05:54:13.468 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2021-07-26T05:54:13.474 INFO:teuthology.orchestra.run.smithi094.stdout:5.4.0-66-generic 2021-07-26T05:54:13.474 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.4.0-66-generic 2021-07-26T05:54:13.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get clean 2021-07-26T05:54:13.550 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get update 2021-07-26T05:54:13.693 INFO:teuthology.orchestra.run.smithi094.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2021-07-26T05:54:13.811 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-07-26T05:54:13.893 INFO:teuthology.orchestra.run.smithi094.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-07-26T05:54:13.972 INFO:teuthology.orchestra.run.smithi094.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [268 kB] 2021-07-26T05:54:14.030 INFO:teuthology.orchestra.run.smithi094.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [789 kB] 2021-07-26T05:54:14.062 INFO:teuthology.orchestra.run.smithi094.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [155 kB] 2021-07-26T05:54:14.066 INFO:teuthology.orchestra.run.smithi094.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [8,304 B] 2021-07-26T05:54:14.066 INFO:teuthology.orchestra.run.smithi094.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [16.5 kB] 2021-07-26T05:54:14.067 INFO:teuthology.orchestra.run.smithi094.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [327 kB] 2021-07-26T05:54:14.070 INFO:teuthology.orchestra.run.smithi094.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [47.4 kB] 2021-07-26T05:54:14.072 INFO:teuthology.orchestra.run.smithi094.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [428 B] 2021-07-26T05:54:14.072 INFO:teuthology.orchestra.run.smithi094.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [505 kB] 2021-07-26T05:54:14.079 INFO:teuthology.orchestra.run.smithi094.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [633 kB] 2021-07-26T05:54:14.086 INFO:teuthology.orchestra.run.smithi094.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [97.5 kB] 2021-07-26T05:54:14.088 INFO:teuthology.orchestra.run.smithi094.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [11.9 kB] 2021-07-26T05:54:14.100 INFO:teuthology.orchestra.run.smithi094.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,912 B] 2021-07-26T05:54:14.101 INFO:teuthology.orchestra.run.smithi094.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.4 kB] 2021-07-26T05:54:14.101 INFO:teuthology.orchestra.run.smithi094.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [4,952 B] 2021-07-26T05:54:14.101 INFO:teuthology.orchestra.run.smithi094.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [600 B] 2021-07-26T05:54:14.225 INFO:teuthology.orchestra.run.smithi094.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-07-26T05:54:14.355 INFO:teuthology.orchestra.run.smithi094.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [511 kB] 2021-07-26T05:54:14.614 INFO:teuthology.orchestra.run.smithi094.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,126 kB] 2021-07-26T05:54:14.739 INFO:teuthology.orchestra.run.smithi094.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [245 kB] 2021-07-26T05:54:14.759 INFO:teuthology.orchestra.run.smithi094.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [13.8 kB] 2021-07-26T05:54:14.760 INFO:teuthology.orchestra.run.smithi094.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [17.8 kB] 2021-07-26T05:54:14.761 INFO:teuthology.orchestra.run.smithi094.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [364 kB] 2021-07-26T05:54:14.785 INFO:teuthology.orchestra.run.smithi094.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [52.7 kB] 2021-07-26T05:54:14.788 INFO:teuthology.orchestra.run.smithi094.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [428 B] 2021-07-26T05:54:14.788 INFO:teuthology.orchestra.run.smithi094.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [626 kB] 2021-07-26T05:54:14.815 INFO:teuthology.orchestra.run.smithi094.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [843 kB] 2021-07-26T05:54:14.848 INFO:teuthology.orchestra.run.smithi094.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [176 kB] 2021-07-26T05:54:14.854 INFO:teuthology.orchestra.run.smithi094.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [18.4 kB] 2021-07-26T05:54:14.871 INFO:teuthology.orchestra.run.smithi094.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.4 kB] 2021-07-26T05:54:14.871 INFO:teuthology.orchestra.run.smithi094.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,912 B] 2021-07-26T05:54:14.872 INFO:teuthology.orchestra.run.smithi094.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [6,780 B] 2021-07-26T05:54:14.906 INFO:teuthology.orchestra.run.smithi094.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [676 B] 2021-07-26T05:54:14.906 INFO:teuthology.orchestra.run.smithi094.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [2,568 B] 2021-07-26T05:54:14.907 INFO:teuthology.orchestra.run.smithi094.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [2,568 B] 2021-07-26T05:54:14.907 INFO:teuthology.orchestra.run.smithi094.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [1,120 B] 2021-07-26T05:54:14.988 INFO:teuthology.orchestra.run.smithi094.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [400 B] 2021-07-26T05:54:14.988 INFO:teuthology.orchestra.run.smithi094.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [4,724 B] 2021-07-26T05:54:14.989 INFO:teuthology.orchestra.run.smithi094.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [5,792 B] 2021-07-26T05:54:14.989 INFO:teuthology.orchestra.run.smithi094.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [2,060 B] 2021-07-26T05:54:14.989 INFO:teuthology.orchestra.run.smithi094.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [288 B] 2021-07-26T05:54:18.021 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 7,282 kB in 2s (4,796 kB/s) 2021-07-26T05:54:19.125 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T05:54:19.147 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-07-26T05:54:19.211 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T05:54:19.370 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-07-26T05:54:19.371 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-07-26T05:54:19.448 INFO:teuthology.orchestra.run.smithi094.stderr:E: Unable to locate package linux-image-current-generic 2021-07-26T05:54:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-07-26T05:54:19.451 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-07-26T05:54:19.518 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T05:54:19.676 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-07-26T05:54:19.677 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-07-26T05:54:19.814 INFO:teuthology.orchestra.run.smithi094.stdout:The following additional packages will be installed: 2021-07-26T05:54:19.815 INFO:teuthology.orchestra.run.smithi094.stdout: linux-generic linux-headers-5.4.0-80 linux-headers-5.4.0-80-generic 2021-07-26T05:54:19.815 INFO:teuthology.orchestra.run.smithi094.stdout: linux-headers-generic linux-image-5.4.0-80-generic 2021-07-26T05:54:19.815 INFO:teuthology.orchestra.run.smithi094.stdout: linux-modules-5.4.0-80-generic linux-modules-extra-5.4.0-80-generic 2021-07-26T05:54:19.816 INFO:teuthology.orchestra.run.smithi094.stdout:Suggested packages: 2021-07-26T05:54:19.816 INFO:teuthology.orchestra.run.smithi094.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2021-07-26T05:54:19.816 INFO:teuthology.orchestra.run.smithi094.stdout:Recommended packages: 2021-07-26T05:54:19.817 INFO:teuthology.orchestra.run.smithi094.stdout: thermald 2021-07-26T05:54:19.844 INFO:teuthology.orchestra.run.smithi094.stdout:The following NEW packages will be installed: 2021-07-26T05:54:19.844 INFO:teuthology.orchestra.run.smithi094.stdout: linux-headers-5.4.0-80 linux-headers-5.4.0-80-generic 2021-07-26T05:54:19.845 INFO:teuthology.orchestra.run.smithi094.stdout: linux-image-5.4.0-80-generic linux-modules-5.4.0-80-generic 2021-07-26T05:54:19.845 INFO:teuthology.orchestra.run.smithi094.stdout: linux-modules-extra-5.4.0-80-generic 2021-07-26T05:54:19.846 INFO:teuthology.orchestra.run.smithi094.stdout:The following packages will be upgraded: 2021-07-26T05:54:19.846 INFO:teuthology.orchestra.run.smithi094.stdout: linux-generic linux-headers-generic linux-image-generic 2021-07-26T05:54:20.027 INFO:teuthology.orchestra.run.smithi094.stdout:3 upgraded, 5 newly installed, 0 to remove and 160 not upgraded. 2021-07-26T05:54:20.027 INFO:teuthology.orchestra.run.smithi094.stdout:Need to get 75.6 MB of archives. 2021-07-26T05:54:20.027 INFO:teuthology.orchestra.run.smithi094.stdout:After this operation, 377 MB of additional disk space will be used. 2021-07-26T05:54:20.028 INFO:teuthology.orchestra.run.smithi094.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-80-generic amd64 5.4.0-80.90 [14.8 MB] 2021-07-26T05:54:21.127 INFO:teuthology.orchestra.run.smithi094.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-80-generic amd64 5.4.0-80.90 [8,976 kB] 2021-07-26T05:54:21.386 INFO:teuthology.orchestra.run.smithi094.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-80-generic amd64 5.4.0-80.90 [39.4 MB] 2021-07-26T05:54:22.734 INFO:teuthology.orchestra.run.smithi094.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.80.84 [1,900 B] 2021-07-26T05:54:22.735 INFO:teuthology.orchestra.run.smithi094.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.80.84 [2,568 B] 2021-07-26T05:54:22.736 INFO:teuthology.orchestra.run.smithi094.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-80 all 5.4.0-80.90 [11.0 MB] 2021-07-26T05:54:23.024 INFO:teuthology.orchestra.run.smithi094.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-80-generic amd64 5.4.0-80.90 [1,410 kB] 2021-07-26T05:54:23.060 INFO:teuthology.orchestra.run.smithi094.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.80.84 [2,436 B] 2021-07-26T05:54:23.455 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 75.6 MB in 3s (23.6 MB/s) 2021-07-26T05:54:23.627 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-modules-5.4.0-80-generic. 2021-07-26T05:54:25.130 INFO:teuthology.orchestra.run.smithi094.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101691 files and directories currently installed.) 2021-07-26T05:54:25.135 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../0-linux-modules-5.4.0-80-generic_5.4.0-80.90_amd64.deb ... 2021-07-26T05:54:25.224 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-modules-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:27.196 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-image-5.4.0-80-generic. 2021-07-26T05:54:27.213 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../1-linux-image-5.4.0-80-generic_5.4.0-80.90_amd64.deb ... 2021-07-26T05:54:27.295 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-image-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:28.299 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-80-generic. 2021-07-26T05:54:28.313 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-80-generic_5.4.0-80.90_amd64.deb ... 2021-07-26T05:54:28.349 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-modules-extra-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:33.362 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../3-linux-generic_5.4.0.80.84_amd64.deb ... 2021-07-26T05:54:33.487 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-generic (5.4.0.80.84) over (5.4.0.66.69) ... 2021-07-26T05:54:33.818 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.80.84_amd64.deb ... 2021-07-26T05:54:33.949 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-image-generic (5.4.0.80.84) over (5.4.0.66.69) ... 2021-07-26T05:54:34.277 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-headers-5.4.0-80. 2021-07-26T05:54:34.290 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../5-linux-headers-5.4.0-80_5.4.0-80.90_all.deb ... 2021-07-26T05:54:34.336 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-headers-5.4.0-80 (5.4.0-80.90) ... 2021-07-26T05:54:41.046 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package linux-headers-5.4.0-80-generic. 2021-07-26T05:54:41.067 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../6-linux-headers-5.4.0-80-generic_5.4.0-80.90_amd64.deb ... 2021-07-26T05:54:41.104 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-headers-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:43.622 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.80.84_amd64.deb ... 2021-07-26T05:54:43.743 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking linux-headers-generic (5.4.0.80.84) over (5.4.0.66.69) ... 2021-07-26T05:54:44.075 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-modules-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:45.265 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-headers-5.4.0-80 (5.4.0-80.90) ... 2021-07-26T05:54:45.390 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-image-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:46.586 INFO:teuthology.orchestra.run.smithi094.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-80-generic 2021-07-26T05:54:46.586 INFO:teuthology.orchestra.run.smithi094.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-80-generic 2021-07-26T05:54:46.720 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-headers-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:46.847 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-modules-extra-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:47.990 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-image-generic (5.4.0.80.84) ... 2021-07-26T05:54:48.091 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-headers-generic (5.4.0.80.84) ... 2021-07-26T05:54:48.192 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up linux-generic (5.4.0.80.84) ... 2021-07-26T05:54:48.319 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for linux-image-5.4.0-80-generic (5.4.0-80.90) ... 2021-07-26T05:54:48.518 INFO:teuthology.orchestra.run.smithi094.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-07-26T05:54:48.518 INFO:teuthology.orchestra.run.smithi094.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-80-generic 2021-07-26T05:55:09.930 INFO:teuthology.orchestra.run.smithi094.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-07-26T05:55:09.931 INFO:teuthology.orchestra.run.smithi094.stdout:Sourcing file `/etc/default/grub' 2021-07-26T05:55:09.935 INFO:teuthology.orchestra.run.smithi094.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-07-26T05:55:09.959 INFO:teuthology.orchestra.run.smithi094.stdout:Generating grub configuration file ... 2021-07-26T05:55:10.279 INFO:teuthology.orchestra.run.smithi094.stdout:Found linux image: /boot/vmlinuz-5.4.0-80-generic 2021-07-26T05:55:10.301 INFO:teuthology.orchestra.run.smithi094.stdout:Found initrd image: /boot/initrd.img-5.4.0-80-generic 2021-07-26T05:55:10.665 INFO:teuthology.orchestra.run.smithi094.stdout:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-07-26T05:55:10.676 INFO:teuthology.orchestra.run.smithi094.stdout:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-07-26T05:55:13.240 INFO:teuthology.orchestra.run.smithi094.stdout:done 2021-07-26T05:55:13.605 DEBUG:teuthology.orchestra.run.smithi094:> dpkg -s linux-image-generic 2021-07-26T05:55:13.629 INFO:teuthology.orchestra.run.smithi094.stdout:Package: linux-image-generic 2021-07-26T05:55:13.629 INFO:teuthology.orchestra.run.smithi094.stdout:Status: install ok installed 2021-07-26T05:55:13.630 INFO:teuthology.orchestra.run.smithi094.stdout:Priority: optional 2021-07-26T05:55:13.630 INFO:teuthology.orchestra.run.smithi094.stdout:Section: kernel 2021-07-26T05:55:13.630 INFO:teuthology.orchestra.run.smithi094.stdout:Installed-Size: 18 2021-07-26T05:55:13.630 INFO:teuthology.orchestra.run.smithi094.stdout:Maintainer: Ubuntu Kernel Team 2021-07-26T05:55:13.630 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: amd64 2021-07-26T05:55:13.631 INFO:teuthology.orchestra.run.smithi094.stdout:Source: linux-meta 2021-07-26T05:55:13.631 INFO:teuthology.orchestra.run.smithi094.stdout:Version: 5.4.0.80.84 2021-07-26T05:55:13.631 INFO:teuthology.orchestra.run.smithi094.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.2), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.10) 2021-07-26T05:55:13.631 INFO:teuthology.orchestra.run.smithi094.stdout:Depends: linux-image-5.4.0-80-generic, linux-modules-extra-5.4.0-80-generic, linux-firmware, intel-microcode, amd64-microcode 2021-07-26T05:55:13.631 INFO:teuthology.orchestra.run.smithi094.stdout:Recommends: thermald 2021-07-26T05:55:13.632 INFO:teuthology.orchestra.run.smithi094.stdout:Description: Generic Linux kernel image 2021-07-26T05:55:13.632 INFO:teuthology.orchestra.run.smithi094.stdout: This package will always depend on the latest generic kernel image 2021-07-26T05:55:13.632 INFO:teuthology.orchestra.run.smithi094.stdout: available. 2021-07-26T05:55:13.633 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-66-generic Expected: 5.4.0-80-generic 2021-07-26T05:55:13.633 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-07-26T05:55:13.634 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-07-26T05:55:13.634 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-07-26T05:55:13.634 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2021-07-26T05:55:13.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get clean 2021-07-26T05:55:13.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get update 2021-07-26T05:55:13.832 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-07-26T05:55:13.951 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-07-26T05:55:14.033 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-07-26T05:55:14.119 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-07-26T05:55:15.303 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T05:55:15.324 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-07-26T05:55:15.385 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T05:55:15.549 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-07-26T05:55:15.550 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-07-26T05:55:15.629 INFO:teuthology.orchestra.run.smithi094.stderr:E: Unable to locate package linux-image-current-generic 2021-07-26T05:55:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-07-26T05:55:15.632 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-07-26T05:55:15.699 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T05:55:15.858 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-07-26T05:55:15.859 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-07-26T05:55:16.011 INFO:teuthology.orchestra.run.smithi094.stdout:linux-image-generic is already the newest version (5.4.0.80.84). 2021-07-26T05:55:16.012 INFO:teuthology.orchestra.run.smithi094.stdout:0 upgraded, 0 newly installed, 0 to remove and 160 not upgraded. 2021-07-26T05:55:16.014 DEBUG:teuthology.orchestra.run.smithi094:> dpkg -s linux-image-generic 2021-07-26T05:55:16.035 INFO:teuthology.orchestra.run.smithi094.stdout:Package: linux-image-generic 2021-07-26T05:55:16.035 INFO:teuthology.orchestra.run.smithi094.stdout:Status: install ok installed 2021-07-26T05:55:16.035 INFO:teuthology.orchestra.run.smithi094.stdout:Priority: optional 2021-07-26T05:55:16.036 INFO:teuthology.orchestra.run.smithi094.stdout:Section: kernel 2021-07-26T05:55:16.036 INFO:teuthology.orchestra.run.smithi094.stdout:Installed-Size: 18 2021-07-26T05:55:16.036 INFO:teuthology.orchestra.run.smithi094.stdout:Maintainer: Ubuntu Kernel Team 2021-07-26T05:55:16.036 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: amd64 2021-07-26T05:55:16.036 INFO:teuthology.orchestra.run.smithi094.stdout:Source: linux-meta 2021-07-26T05:55:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:Version: 5.4.0.80.84 2021-07-26T05:55:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.2), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.10) 2021-07-26T05:55:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:Depends: linux-image-5.4.0-80-generic, linux-modules-extra-5.4.0-80-generic, linux-firmware, intel-microcode, amd64-microcode 2021-07-26T05:55:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:Recommends: thermald 2021-07-26T05:55:16.037 INFO:teuthology.orchestra.run.smithi094.stdout:Description: Generic Linux kernel image 2021-07-26T05:55:16.038 INFO:teuthology.orchestra.run.smithi094.stdout: This package will always depend on the latest generic kernel image 2021-07-26T05:55:16.038 INFO:teuthology.orchestra.run.smithi094.stdout: available. 2021-07-26T05:55:16.038 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2021-07-26T05:55:16.043 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.JHSi6ETGT7 2021-07-26T05:55:16.044 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub/grub.cfg /tmp/tmp.JHSi6ETGT7 2021-07-26T05:55:16.053 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.JHSi6ETGT7 2021-07-26T05:55:16.138 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.JHSi6ETGT7 is 8KB 2021-07-26T05:55:16.152 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.JHSi6ETGT7 2021-07-26T05:55:16.159 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-07-26T05:55:16.215 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-07-26T05:55:16.215 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-07-26T05:55:16.216 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-07-26T05:55:16.243 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-80-generic 2021-07-26T05:55:16.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo update-grub 2021-07-26T05:55:17.418 INFO:teuthology.orchestra.run.smithi094.stderr:Sourcing file `/etc/default/grub' 2021-07-26T05:55:17.420 INFO:teuthology.orchestra.run.smithi094.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-07-26T05:55:17.433 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2021-07-26T05:55:17.671 INFO:teuthology.orchestra.run.smithi094.stderr:Found linux image: /boot/vmlinuz-5.4.0-80-generic 2021-07-26T05:55:17.682 INFO:teuthology.orchestra.run.smithi094.stderr:Found initrd image: /boot/initrd.img-5.4.0-80-generic 2021-07-26T05:55:18.009 INFO:teuthology.orchestra.run.smithi094.stderr:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-07-26T05:55:18.020 INFO:teuthology.orchestra.run.smithi094.stderr:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-07-26T05:55:19.568 INFO:teuthology.orchestra.run.smithi094.stderr:done 2021-07-26T05:55:19.570 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2021-07-26T05:55:49.603 INFO:teuthology.misc:Re-opening connections... 2021-07-26T05:55:49.604 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-07-26T05:55:49.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T05:55:49.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:56:49.657 DEBUG:teuthology.orchestra.remote:timed out 2021-07-26T05:56:49.658 DEBUG:teuthology.misc:waited 60.05478858947754 2021-07-26T05:56:50.660 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-07-26T05:56:50.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T05:56:50.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:56:51.119 DEBUG:teuthology.orchestra.remote:No existing session 2021-07-26T05:56:51.120 DEBUG:teuthology.misc:waited 61.516528367996216 2021-07-26T05:56:52.122 INFO:teuthology.misc:trying to connect to ubuntu@smithi094.front.sepia.ceph.com 2021-07-26T05:56:52.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T05:56:52.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T05:56:52.156 DEBUG:teuthology.orchestra.run.smithi094:> true 2021-07-26T05:56:52.872 DEBUG:teuthology.misc:waited 63.26865363121033 2021-07-26T05:56:53.873 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-07-26T05:56:53.874 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-80-generic"... 2021-07-26T05:56:53.874 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2021-07-26T05:56:53.881 INFO:teuthology.orchestra.run.smithi094.stdout:5.4.0-80-generic 2021-07-26T05:56:53.882 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-80-generic vs 5.4.0-80-generic 2021-07-26T05:56:53.882 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-07-26T05:56:53.882 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-07-26T05:56:54.884 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-07-26T05:56:54.884 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-07-26T05:56:54.962 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2021-07-26T05:56:54.981 INFO:teuthology.run_tasks:Running task internal.base... 2021-07-26T05:56:54.990 INFO:teuthology.task.internal:Creating test directory... 2021-07-26T05:56:54.991 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-07-26T05:56:54.998 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-07-26T05:56:55.081 INFO:teuthology.run_tasks:Running task internal.archive... 2021-07-26T05:56:55.173 INFO:teuthology.task.internal:Creating archive directory... 2021-07-26T05:56:55.173 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-07-26T05:56:55.202 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-07-26T05:56:55.297 INFO:teuthology.task.internal:Enabling coredump saving... 2021-07-26T05:56:55.298 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-07-26T05:56:55.337 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T05:56:55.347 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T05:56:55.349 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-07-26T05:56:55.407 INFO:teuthology.task.internal:Configuring sudo... 2021-07-26T05:56:55.407 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-07-26T05:56:55.423 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-07-26T05:56:55.466 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-07-26T05:56:55.466 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-07-26T05:56:55.472 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-07-26T05:56:55.521 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-07-26T05:56:55.528 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-07-26T05:56:55.528 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-07-26T05:56:55.556 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2021-07-26T05:56:55.603 INFO:teuthology.run_tasks:Running task internal.timer... 2021-07-26T05:56:55.614 INFO:teuthology.task.internal:Starting timer... 2021-07-26T05:56:55.615 INFO:teuthology.run_tasks:Running task pcp... 2021-07-26T05:56:55.700 INFO:teuthology.run_tasks:Running task selinux... 2021-07-26T05:56:55.790 DEBUG:teuthology.task.selinux:Excluding smithi094: OS 'ubuntu' does not support SELinux 2021-07-26T05:56:55.791 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-07-26T05:56:55.791 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-07-26T05:56:55.791 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-07-26T05:56:55.792 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-07-26T05:56:55.907 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-07-26T05:56:55.907 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-07-26T05:56:55.927 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-07-26T05:56:55.929 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-07-26T05:56:56.250 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-07-26T05:56:56.981 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-07-26T05:56:59.618 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-07-26T05:56:59.639 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:56:59 +0000 (0:00:00.035) 0:00:00.035 *********** 2021-07-26T05:57:01.825 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-07-26T05:57:01.827 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi094.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi094.front.sepia.ceph.com] 2021-07-26T05:57:01.873 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:01 +0000 (0:00:02.233) 0:00:02.269 *********** 2021-07-26T05:57:02.258 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-07-26T05:57:02.260 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-07-26T05:57:02.280 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:02 +0000 (0:00:00.407) 0:00:02.676 *********** 2021-07-26T05:57:03.049 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-07-26T05:57:03.051 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-07-26T05:57:03.072 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:03 +0000 (0:00:00.791) 0:00:03.468 *********** 2021-07-26T05:57:03.427 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-07-26T05:57:03.429 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.061167", "end": "2021-07-26 05:57:02.918153", "rc": 0, "start": "2021-07-26 05:57:02.856986", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-07-26T05:57:03.450 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:03 +0000 (0:00:00.378) 0:00:03.846 *********** 2021-07-26T05:57:03.691 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-07-26T05:57:03.692 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:57:03.713 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:03 +0000 (0:00:00.263) 0:00:04.109 *********** 2021-07-26T05:57:04.613 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-07-26T05:57:04.615 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279023.7471657-26757-162620946632178/source", "state": "file", "uid": 0} 2021-07-26T05:57:04.635 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:04 +0000 (0:00:00.921) 0:00:05.031 *********** 2021-07-26T05:57:05.052 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-07-26T05:57:05.054 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-07-26T05:57:05.055 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-07-26T05:57:05.117 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-07-26T05:57:05.119 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-26T05:57:05.132 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Monday 26 July 2021 05:57:05 +0000 (0:00:00.496) 0:00:05.528 *********** 2021-07-26T05:57:05.816 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] 2021-07-26T05:57:05.833 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Monday 26 July 2021 05:57:05 +0000 (0:00:00.701) 0:00:06.229 *********** 2021-07-26T05:57:05.860 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-07-26T05:57:05.864 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-26T05:57:06.015 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:06 +0000 (0:00:00.181) 0:00:06.411 *********** 2021-07-26T05:57:06.825 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi094.front.sepia.ceph.com] 2021-07-26T05:57:06.876 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-07-26T05:57:06.877 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:06 +0000 (0:00:00.860) 0:00:07.272 *********** 2021-07-26T05:57:07.274 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-07-26T05:57:07.274 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.174529", "end": "2021-07-26 05:57:07.254220", "rc": 0, "start": "2021-07-26 05:57:07.079691", "stderr": "", "stderr_lines": [], "stdout": "fa3530b6c8007ac148c609947f616c9ce81a7370", "stdout_lines": ["fa3530b6c8007ac148c609947f616c9ce81a7370"]} 2021-07-26T05:57:07.295 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:07 +0000 (0:00:00.419) 0:00:07.691 *********** 2021-07-26T05:57:07.455 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-07-26T05:57:07.457 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.024799", "end": "2021-07-26 05:57:06.949814", "failed_when_result": false, "rc": 0, "start": "2021-07-26 05:57:06.925015", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-07-26T05:57:07.477 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:07 +0000 (0:00:00.181) 0:00:07.873 *********** 2021-07-26T05:57:07.544 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-07-26T05:57:07.546 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:07.566 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:07 +0000 (0:00:00.089) 0:00:07.962 *********** 2021-07-26T05:57:07.623 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-07-26T05:57:07.624 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:07.644 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:07 +0000 (0:00:00.078) 0:00:08.040 *********** 2021-07-26T05:57:07.700 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-07-26T05:57:07.701 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:07.721 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:07 +0000 (0:00:00.077) 0:00:08.118 *********** 2021-07-26T05:57:07.787 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-07-26T05:57:07.789 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:07.809 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:07 +0000 (0:00:00.087) 0:00:08.205 *********** 2021-07-26T05:57:07.876 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-07-26T05:57:07.878 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:07.898 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:07 +0000 (0:00:00.088) 0:00:08.294 *********** 2021-07-26T05:57:08.078 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-07-26T05:57:08.080 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:57:08.100 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:08 +0000 (0:00:00.202) 0:00:08.497 *********** 2021-07-26T05:57:08.332 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-07-26T05:57:08.334 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-07-26T05:57:08.471 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-07-26T05:57:08.616 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-07-26T05:57:08.750 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-07-26T05:57:08.885 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-07-26T05:57:09.030 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-07-26T05:57:09.164 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-07-26T05:57:09.303 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-07-26T05:57:09.437 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-07-26T05:57:09.581 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-07-26T05:57:09.716 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-07-26T05:57:09.861 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-07-26T05:57:09.996 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-07-26T05:57:10.142 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-07-26T05:57:10.277 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-07-26T05:57:10.423 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-07-26T05:57:10.558 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-07-26T05:57:10.693 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-07-26T05:57:10.839 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-07-26T05:57:10.974 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-07-26T05:57:11.119 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-07-26T05:57:11.255 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-07-26T05:57:11.399 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-07-26T05:57:11.544 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-07-26T05:57:11.678 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-07-26T05:57:11.813 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-07-26T05:57:11.959 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-07-26T05:57:12.094 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-07-26T05:57:12.243 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-07-26T05:57:12.380 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-07-26T05:57:12.526 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-07-26T05:57:12.673 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-07-26T05:57:12.809 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-07-26T05:57:12.946 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-07-26T05:57:13.082 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-07-26T05:57:13.228 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-07-26T05:57:13.363 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-07-26T05:57:13.519 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-26T05:57:13.655 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-07-26T05:57:13.801 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-07-26T05:57:13.936 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-07-26T05:57:14.081 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-07-26T05:57:14.216 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-07-26T05:57:14.361 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-07-26T05:57:14.496 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-07-26T05:57:14.641 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-26T05:57:14.777 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-07-26T05:57:14.922 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-07-26T05:57:15.068 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-07-26T05:57:15.203 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-07-26T05:57:15.338 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-07-26T05:57:15.484 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-07-26T05:57:15.618 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-07-26T05:57:15.764 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-07-26T05:57:15.900 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-07-26T05:57:16.045 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-07-26T05:57:16.181 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-07-26T05:57:16.316 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-07-26T05:57:16.461 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-07-26T05:57:16.597 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-07-26T05:57:16.744 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-07-26T05:57:16.880 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-07-26T05:57:17.015 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-07-26T05:57:17.161 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-07-26T05:57:17.296 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-07-26T05:57:17.442 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-07-26T05:57:17.588 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-07-26T05:57:17.723 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-07-26T05:57:17.869 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-07-26T05:57:18.014 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-07-26T05:57:18.149 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-07-26T05:57:18.294 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-07-26T05:57:18.429 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-07-26T05:57:18.574 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-07-26T05:57:18.708 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-07-26T05:57:18.854 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-07-26T05:57:19.000 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-07-26T05:57:19.135 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-07-26T05:57:19.270 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-07-26T05:57:19.415 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-07-26T05:57:19.550 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-07-26T05:57:19.695 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-07-26T05:57:19.830 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-07-26T05:57:19.965 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-07-26T05:57:20.110 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-07-26T05:57:20.245 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-07-26T05:57:20.390 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-07-26T05:57:20.525 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-07-26T05:57:20.659 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-07-26T05:57:20.815 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-07-26T05:57:20.949 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-07-26T05:57:21.084 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-07-26T05:57:21.229 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-07-26T05:57:21.363 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-07-26T05:57:21.508 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-07-26T05:57:21.643 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-07-26T05:57:21.788 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-07-26T05:57:21.923 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-07-26T05:57:22.070 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-07-26T05:57:22.215 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-07-26T05:57:22.349 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-07-26T05:57:22.494 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-07-26T05:57:22.629 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-07-26T05:57:22.764 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-07-26T05:57:22.900 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-07-26T05:57:23.046 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-07-26T05:57:23.191 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-07-26T05:57:23.327 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-07-26T05:57:23.464 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-07-26T05:57:23.599 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-07-26T05:57:23.745 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-07-26T05:57:23.880 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-07-26T05:57:24.015 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-07-26T05:57:24.160 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-07-26T05:57:24.295 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-07-26T05:57:24.441 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-07-26T05:57:24.586 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-07-26T05:57:24.731 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-07-26T05:57:24.866 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-07-26T05:57:25.011 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-07-26T05:57:25.156 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-07-26T05:57:25.291 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-07-26T05:57:25.427 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-07-26T05:57:25.572 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-07-26T05:57:25.707 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-07-26T05:57:25.842 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-07-26T05:57:25.989 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-07-26T05:57:26.123 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-07-26T05:57:26.269 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-07-26T05:57:26.404 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-07-26T05:57:26.538 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-07-26T05:57:26.683 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-07-26T05:57:26.818 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-07-26T05:57:26.962 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-07-26T05:57:27.096 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-07-26T05:57:27.242 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-07-26T05:57:27.376 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-07-26T05:57:27.521 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-07-26T05:57:28.047 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-07-26T05:57:28.594 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-07-26T05:57:29.141 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-07-26T05:57:29.585 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-07-26T05:57:30.041 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-07-26T05:57:30.505 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-07-26T05:57:31.010 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-07-26T05:57:31.505 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-07-26T05:57:32.032 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-07-26T05:57:32.557 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-07-26T05:57:33.032 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-07-26T05:57:33.505 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-07-26T05:57:34.042 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-07-26T05:57:34.546 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-07-26T05:57:35.030 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-07-26T05:57:35.142 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:35 +0000 (0:00:27.041) 0:00:35.538 *********** 2021-07-26T05:57:35.210 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-07-26T05:57:35.232 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:35 +0000 (0:00:00.090) 0:00:35.628 *********** 2021-07-26T05:57:35.321 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-07-26T05:57:35.323 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-07-26T05:57:35.325 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-07-26T05:57:35.327 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-07-26T05:57:35.328 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-07-26T05:57:35.330 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99 2021-07-26T05:57:35.330 INFO:teuthology.task.ansible.out:bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-07-26T05:57:35.332 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-07-26T05:57:35.333 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-07-26T05:57:35.335 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-07-26T05:57:35.337 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-07-26T05:57:35.338 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-07-26T05:57:35.340 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-07-26T05:57:35.342 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0 2021-07-26T05:57:35.342 INFO:teuthology.task.ansible.out:fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-07-26T05:57:35.344 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-07-26T05:57:35.345 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}]}, "changed": false} 2021-07-26T05:57:35.369 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:35 +0000 (0:00:00.136) 0:00:35.765 *********** 2021-07-26T05:57:36.260 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-07-26T05:57:36.261 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"after": "fa3530b6c8007ac148c609947f616c9ce81a7370", "attempts": 1, "before": "fa3530b6c8007ac148c609947f616c9ce81a7370", "changed": false, "remote_url_changed": false} 2021-07-26T05:57:36.283 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:36 +0000 (0:00:00.914) 0:00:36.679 *********** 2021-07-26T05:57:36.527 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-07-26T05:57:36.529 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-07-26T05:57:36.652 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-07-26T05:57:36.777 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-07-26T05:57:36.911 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-07-26T05:57:36.913 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-07-26T05:57:37.046 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-07-26T05:57:37.181 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-07-26T05:57:37.183 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-07-26T05:57:37.306 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-07-26T05:57:37.441 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-07-26T05:57:37.565 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-07-26T05:57:37.576 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:37.703 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-07-26T05:57:37.704 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-07-26T05:57:37.827 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-07-26T05:57:37.829 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-07-26T05:57:37.838 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:37.849 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:37.973 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-07-26T05:57:38.108 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-07-26T05:57:38.110 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-07-26T05:57:38.232 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-07-26T05:57:38.357 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-07-26T05:57:38.359 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-07-26T05:57:38.502 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-07-26T05:57:38.626 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-07-26T05:57:38.750 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-07-26T05:57:38.874 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-07-26T05:57:38.876 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-07-26T05:57:39.008 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-07-26T05:57:39.010 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-07-26T05:57:39.143 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-07-26T05:57:39.277 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-07-26T05:57:39.401 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-07-26T05:57:39.403 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-07-26T05:57:39.526 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-07-26T05:57:39.661 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-07-26T05:57:39.785 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-07-26T05:57:39.919 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-07-26T05:57:40.064 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-07-26T05:57:40.208 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-07-26T05:57:40.210 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-07-26T05:57:40.333 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-07-26T05:57:40.467 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-07-26T05:57:40.602 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-07-26T05:57:40.726 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-07-26T05:57:40.850 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-07-26T05:57:40.974 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-26T05:57:41.099 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-07-26T05:57:41.234 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-07-26T05:57:41.358 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-07-26T05:57:41.360 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-07-26T05:57:41.492 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-07-26T05:57:41.494 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-07-26T05:57:41.617 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-07-26T05:57:41.741 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-07-26T05:57:41.865 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-07-26T05:57:42.000 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtV 2021-07-26T05:57:42.002 INFO:teuthology.task.ansible.out:TFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-26T05:57:42.137 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-07-26T05:57:42.139 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-07-26T05:57:42.140 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-07-26T05:57:42.261 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-07-26T05:57:42.396 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-07-26T05:57:42.519 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-07-26T05:57:42.654 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-07-26T05:57:42.779 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-07-26T05:57:42.903 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-07-26T05:57:42.905 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-07-26T05:57:43.038 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-07-26T05:57:43.162 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-07-26T05:57:43.297 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-07-26T05:57:43.299 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-07-26T05:57:43.421 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-07-26T05:57:43.546 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-07-26T05:57:43.681 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-07-26T05:57:43.681 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-07-26T05:57:43.805 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-07-26T05:57:43.930 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-07-26T05:57:44.064 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-07-26T05:57:44.188 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-07-26T05:57:44.323 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-07-26T05:57:44.467 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-07-26T05:57:44.592 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-07-26T05:57:44.726 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-07-26T05:57:44.850 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-07-26T05:57:44.974 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-07-26T05:57:45.108 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-07-26T05:57:45.233 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-07-26T05:57:45.357 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-07-26T05:57:45.503 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-07-26T05:57:45.637 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-07-26T05:57:45.772 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-07-26T05:57:45.907 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-07-26T05:57:46.031 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-07-26T05:57:46.168 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-07-26T05:57:46.292 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-07-26T05:57:46.437 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-07-26T05:57:46.439 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-07-26T05:57:46.562 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-07-26T05:57:46.697 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-07-26T05:57:46.821 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-07-26T05:57:46.946 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-07-26T05:57:47.081 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-07-26T05:57:47.082 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-07-26T05:57:47.225 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-07-26T05:57:47.227 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-07-26T05:57:47.350 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-07-26T05:57:47.475 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-07-26T05:57:47.600 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-07-26T05:57:47.724 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-07-26T05:57:47.859 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-07-26T05:57:47.994 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-07-26T05:57:48.119 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-07-26T05:57:48.256 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-07-26T05:57:48.258 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-07-26T05:57:48.381 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-07-26T05:57:48.506 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-07-26T05:57:48.641 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-07-26T05:57:48.765 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-07-26T05:57:48.889 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-07-26T05:57:49.023 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-07-26T05:57:49.148 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-07-26T05:57:49.149 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-07-26T05:57:49.282 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-07-26T05:57:49.407 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-07-26T05:57:49.542 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-07-26T05:57:49.676 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-07-26T05:57:49.801 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-07-26T05:57:49.935 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-07-26T05:57:50.101 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-07-26T05:57:50.236 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-07-26T05:57:50.361 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-07-26T05:57:50.496 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-07-26T05:57:50.620 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-07-26T05:57:50.755 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-07-26T05:57:50.879 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-07-26T05:57:51.014 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-07-26T05:57:51.138 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-07-26T05:57:51.263 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-07-26T05:57:51.398 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-07-26T05:57:51.532 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-07-26T05:57:51.657 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-07-26T05:57:51.658 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-07-26T05:57:51.791 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-07-26T05:57:51.926 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-07-26T05:57:52.050 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-07-26T05:57:52.195 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-07-26T05:57:52.319 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-07-26T05:57:52.454 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-07-26T05:57:52.589 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-07-26T05:57:52.724 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-07-26T05:57:52.848 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-07-26T05:57:52.982 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-07-26T05:57:53.117 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-07-26T05:57:53.241 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-07-26T05:57:53.375 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-07-26T05:57:53.509 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-07-26T05:57:53.634 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-07-26T05:57:53.769 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-07-26T05:57:53.903 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-07-26T05:57:53.905 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-07-26T05:57:54.028 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-07-26T05:57:54.152 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-07-26T05:57:54.287 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-07-26T05:57:54.413 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-07-26T05:57:54.537 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-07-26T05:57:54.661 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-07-26T05:57:54.796 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-07-26T05:57:54.919 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-07-26T05:57:55.044 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-07-26T05:57:55.169 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-07-26T05:57:55.294 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-07-26T05:57:55.439 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICYMwYvykHtnVhhJwcKaiRh9oMpJkqvEA7OAbHPYOv6W matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-07-26T05:57:55.562 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-07-26T05:57:55.687 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-07-26T05:57:55.812 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-07-26T05:57:55.946 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-07-26T05:57:56.025 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:56 +0000 (0:00:19.741) 0:00:56.421 *********** 2021-07-26T05:57:56.102 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-07-26T05:57:56.104 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.114 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.115 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.125 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.127 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.136 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.138 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.148 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.159 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.294 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-07-26T05:57:56.305 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.306 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.440 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-07-26T05:57:56.565 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-07-26T05:57:56.576 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.577 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.587 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.588 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.598 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.609 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.610 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.620 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.622 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.631 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.633 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.643 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.653 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.655 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.665 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.666 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.676 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.677 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.687 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.698 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.699 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.709 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.710 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.720 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.722 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.731 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.742 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.744 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.754 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.755 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.765 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.766 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.777 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.788 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.790 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.799 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.801 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.811 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.812 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.822 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.824 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.834 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.845 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.846 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.856 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.858 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.867 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.869 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.878 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.880 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.890 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.890 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.901 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.912 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.914 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.923 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.925 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.935 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.936 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.946 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.957 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.958 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.968 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.969 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.979 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.981 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.990 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:56.991 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.002 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.012 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.014 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.024 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.025 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.035 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.037 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.046 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.057 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.059 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.068 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.070 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.079 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.081 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.091 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.092 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.102 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.113 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.115 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.125 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.126 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.136 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.138 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.147 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.149 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.158 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.169 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.171 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.181 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.182 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.192 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.194 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.203 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.205 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.214 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.225 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.226 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.237 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.238 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.248 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.249 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.259 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.270 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.271 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.281 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.282 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.292 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.293 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.303 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.315 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.317 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.326 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.328 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.337 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.339 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.348 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.360 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.362 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.371 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.373 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.383 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.384 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.394 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.396 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.405 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.416 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.418 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matanb@main 5ZJUAUM7LCpjmIkH+Ntl6g 27de66609fee4975af229f411a1f49f589a4c64b958d8913d3f5eff128a37afa"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.428 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.429 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.439 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.450 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.489 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:57 +0000 (0:00:01.464) 0:00:57.885 *********** 2021-07-26T05:57:57.567 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-07-26T05:57:57.569 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:57:57.589 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:57:57 +0000 (0:00:00.100) 0:00:57.985 *********** 2021-07-26T05:57:57.791 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-07-26T05:57:57.792 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-07-26T05:57:57.926 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-07-26T05:57:58.062 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-07-26T05:57:58.198 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-07-26T05:57:58.327 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-07-26T05:57:58.463 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-07-26T05:57:58.589 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-07-26T05:57:58.723 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-07-26T05:57:58.848 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-07-26T05:57:58.984 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-07-26T05:57:59.109 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-07-26T05:57:59.244 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-07-26T05:57:59.379 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-07-26T05:57:59.513 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-07-26T05:57:59.638 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-07-26T05:57:59.772 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-07-26T05:57:59.917 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-07-26T05:58:00.042 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-07-26T05:58:00.177 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-07-26T05:58:00.302 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-07-26T05:58:00.436 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-07-26T05:58:00.571 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-07-26T05:58:00.706 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-07-26T05:58:00.840 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-07-26T05:58:00.975 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-07-26T05:58:01.110 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-07-26T05:58:01.245 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-07-26T05:58:01.379 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-07-26T05:58:01.504 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-07-26T05:58:01.639 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-07-26T05:58:01.774 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-07-26T05:58:01.899 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-07-26T05:58:02.034 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-07-26T05:58:02.169 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-07-26T05:58:02.298 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-07-26T05:58:02.432 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-07-26T05:58:02.567 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-07-26T05:58:02.691 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-07-26T05:58:03.124 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-07-26T05:58:03.259 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-07-26T05:58:03.394 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-07-26T05:58:03.528 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-07-26T05:58:03.653 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-07-26T05:58:03.787 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-07-26T05:58:03.922 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-07-26T05:58:04.045 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-07-26T05:58:04.180 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-07-26T05:58:04.317 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-07-26T05:58:04.442 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-07-26T05:58:04.497 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:04 +0000 (0:00:06.907) 0:01:04.893 *********** 2021-07-26T05:58:04.896 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-07-26T05:58:04.898 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "6b66beacdb1c95b0ac86e1041f3914cfa0c50311", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "e34309bf71492baa6798d79247cca3d0", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279084.5808175-28895-131989626258760/source", "state": "file", "uid": 0} 2021-07-26T05:58:04.920 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:04 +0000 (0:00:00.422) 0:01:05.316 *********** 2021-07-26T05:58:04.987 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-07-26T05:58:04.989 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => { "msg": "Host smithi094.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-07-26T05:58:05.011 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.091) 0:01:05.407 *********** 2021-07-26T05:58:05.089 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-07-26T05:58:05.091 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-07-26T05:58:05.112 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.101) 0:01:05.508 *********** 2021-07-26T05:58:05.180 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-07-26T05:58:05.182 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.210 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.097) 0:01:05.606 *********** 2021-07-26T05:58:05.270 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-07-26T05:58:05.271 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.298 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.087) 0:01:05.694 *********** 2021-07-26T05:58:05.365 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-07-26T05:58:05.367 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.389 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.091) 0:01:05.785 *********** 2021-07-26T05:58:05.457 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-07-26T05:58:05.459 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.480 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.091) 0:01:05.876 *********** 2021-07-26T05:58:05.547 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-07-26T05:58:05.549 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.581 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.100) 0:01:05.977 *********** 2021-07-26T05:58:05.648 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-07-26T05:58:05.650 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.671 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.090) 0:01:06.067 *********** 2021-07-26T05:58:05.738 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-07-26T05:58:05.740 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.762 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.090) 0:01:06.158 *********** 2021-07-26T05:58:05.830 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-07-26T05:58:05.832 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:05.853 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.091) 0:01:06.249 *********** 2021-07-26T05:58:05.931 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-07-26T05:58:05.934 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:58:05.958 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:05 +0000 (0:00:00.104) 0:01:06.354 *********** 2021-07-26T05:58:06.035 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-07-26T05:58:06.037 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.059 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.100) 0:01:06.455 *********** 2021-07-26T05:58:06.126 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-07-26T05:58:06.128 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.150 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.091) 0:01:06.546 *********** 2021-07-26T05:58:06.218 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-07-26T05:58:06.220 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.241 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.091) 0:01:06.637 *********** 2021-07-26T05:58:06.309 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-07-26T05:58:06.311 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:58:06.332 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.090) 0:01:06.728 *********** 2021-07-26T05:58:06.400 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-07-26T05:58:06.402 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.433 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.100) 0:01:06.829 *********** 2021-07-26T05:58:06.502 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-07-26T05:58:06.504 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:58:06.525 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.092) 0:01:06.921 *********** 2021-07-26T05:58:06.604 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-07-26T05:58:06.606 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.627 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.101) 0:01:07.023 *********** 2021-07-26T05:58:06.705 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-07-26T05:58:06.707 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.731 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.103) 0:01:07.127 *********** 2021-07-26T05:58:06.811 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-07-26T05:58:06.813 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.834 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.103) 0:01:07.230 *********** 2021-07-26T05:58:06.902 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-07-26T05:58:06.904 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:06.926 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:06 +0000 (0:00:00.092) 0:01:07.322 *********** 2021-07-26T05:58:06.995 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-07-26T05:58:06.997 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:58:07.027 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.100) 0:01:07.423 *********** 2021-07-26T05:58:07.097 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-26T05:58:07.099 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.127 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.099) 0:01:07.523 *********** 2021-07-26T05:58:07.195 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-07-26T05:58:07.197 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.218 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.091) 0:01:07.614 *********** 2021-07-26T05:58:07.297 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-07-26T05:58:07.299 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.322 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.103) 0:01:07.718 *********** 2021-07-26T05:58:07.380 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-07-26T05:58:07.382 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.404 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.081) 0:01:07.800 *********** 2021-07-26T05:58:07.471 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-07-26T05:58:07.474 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.495 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.091) 0:01:07.891 *********** 2021-07-26T05:58:07.563 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-07-26T05:58:07.565 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.586 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.091) 0:01:07.982 *********** 2021-07-26T05:58:07.676 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-07-26T05:58:07.679 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:58:07.704 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.117) 0:01:08.100 *********** 2021-07-26T05:58:07.784 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-07-26T05:58:07.786 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.808 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.103) 0:01:08.204 *********** 2021-07-26T05:58:07.876 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-07-26T05:58:07.878 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.901 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.093) 0:01:08.297 *********** 2021-07-26T05:58:07.970 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-07-26T05:58:07.972 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:07.996 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:07 +0000 (0:00:00.094) 0:01:08.392 *********** 2021-07-26T05:58:08.075 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-07-26T05:58:08.077 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:08.104 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:08 +0000 (0:00:00.108) 0:01:08.500 *********** 2021-07-26T05:58:08.183 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi094.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:08.186 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:08.207 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:08 +0000 (0:00:00.102) 0:01:08.603 *********** 2021-07-26T05:58:08.296 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-07-26T05:58:08.298 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:08.324 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:08 +0000 (0:00:00.116) 0:01:08.720 *********** 2021-07-26T05:58:08.517 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-26T05:58:08.519 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.010390", "end": "2021-07-26 05:58:08.007263", "rc": 0, "start": "2021-07-26 05:58:07.996873", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-07-26T05:58:08.541 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:08 +0000 (0:00:00.217) 0:01:08.937 *********** 2021-07-26T05:58:08.608 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-07-26T05:58:08.610 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:08.631 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:08 +0000 (0:00:00.090) 0:01:09.027 *********** 2021-07-26T05:58:08.709 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-07-26T05:58:08.711 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:08.732 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:08 +0000 (0:00:00.101) 0:01:09.128 *********** 2021-07-26T05:58:09.421 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-07-26T05:58:09.423 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-07-26T05:58:09.599 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-07-26T05:58:09.778 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-07-26T05:58:09.944 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-07-26T05:58:09.970 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:09 +0000 (0:00:01.237) 0:01:10.366 *********** 2021-07-26T05:58:10.038 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-07-26T05:58:10.040 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:10.065 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:10 +0000 (0:00:00.095) 0:01:10.461 *********** 2021-07-26T05:58:10.144 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-07-26T05:58:10.146 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:10.167 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:10 +0000 (0:00:00.102) 0:01:10.563 *********** 2021-07-26T05:58:10.235 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-07-26T05:58:10.237 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:10.259 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:10 +0000 (0:00:00.091) 0:01:10.655 *********** 2021-07-26T05:58:10.327 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-07-26T05:58:10.329 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:10.353 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:10 +0000 (0:00:00.094) 0:01:10.749 *********** 2021-07-26T05:58:10.422 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-07-26T05:58:10.424 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:10.446 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:10 +0000 (0:00:00.092) 0:01:10.842 *********** 2021-07-26T05:58:11.009 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-07-26T05:58:11.011 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.433856", "end": "2021-07-26 05:58:10.504739", "rc": 0, "start": "2021-07-26 05:58:10.070883", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T05:58:11.033 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:11 +0000 (0:00:00.587) 0:01:11.429 *********** 2021-07-26T05:58:11.101 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-07-26T05:58:11.102 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:11.124 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:11 +0000 (0:00:00.090) 0:01:11.520 *********** 2021-07-26T05:58:11.410 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-07-26T05:58:11.411 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.106616", "end": "2021-07-26 05:58:10.893261", "rc": 0, "start": "2021-07-26 05:58:10.786645", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T05:58:11.433 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:11 +0000 (0:00:00.309) 0:01:11.829 *********** 2021-07-26T05:58:15.415 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-07-26T05:58:15.417 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:58:15.438 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:15 +0000 (0:00:04.005) 0:01:15.834 *********** 2021-07-26T05:58:16.148 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-07-26T05:58:16.149 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:58:16.171 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:16 +0000 (0:00:00.732) 0:01:16.567 *********** 2021-07-26T05:58:16.239 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-07-26T05:58:16.240 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:16.262 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:16 +0000 (0:00:00.090) 0:01:16.658 *********** 2021-07-26T05:58:16.914 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-07-26T05:58:16.916 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-07-26T05:58:16.937 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:16 +0000 (0:00:00.674) 0:01:17.333 *********** 2021-07-26T05:58:17.649 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-07-26T05:58:17.650 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:58:17.672 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:17 +0000 (0:00:00.734) 0:01:18.068 *********** 2021-07-26T05:58:18.070 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-07-26T05:58:18.072 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-07-26T05:58:18.394 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-07-26T05:58:18.695 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-07-26T05:58:18.720 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:18 +0000 (0:00:01.047) 0:01:19.116 *********** 2021-07-26T05:58:18.861 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-07-26T05:58:18.863 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T05:58:18.886 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:18 +0000 (0:00:00.165) 0:01:19.282 *********** 2021-07-26T05:58:19.255 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-07-26T05:58:19.257 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-07-26T05:58:19.545 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-07-26T05:58:19.889 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "fae049f2bb3feda9c19a2dc30bce838a", "mode": "0755", "owner": "root", "size": 3612, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279099.5577264-30115-197098101885243/source", "state": "file", "uid": 0} 2021-07-26T05:58:19.913 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:19 +0000 (0:00:01.027) 0:01:20.309 *********** 2021-07-26T05:58:19.994 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-07-26T05:58:19.995 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-07-26T05:58:20.019 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:20 +0000 (0:00:00.105) 0:01:20.415 *********** 2021-07-26T05:58:20.088 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:20.111 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:20 +0000 (0:00:00.091) 0:01:20.507 *********** 2021-07-26T05:58:20.179 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-07-26T05:58:20.181 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:58:20.203 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:20 +0000 (0:00:00.091) 0:01:20.599 *********** 2021-07-26T05:58:20.282 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-07-26T05:58:20.284 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:20.295 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:20.451 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:20.462 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:20.486 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:20 +0000 (0:00:00.282) 0:01:20.882 *********** 2021-07-26T05:58:20.711 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-07-26T05:58:20.713 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:20.735 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:20 +0000 (0:00:00.248) 0:01:21.131 *********** 2021-07-26T05:58:21.582 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:58:21.605 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:21 +0000 (0:00:00.870) 0:01:22.001 *********** 2021-07-26T05:58:21.982 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-07-26T05:58:22.006 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:22 +0000 (0:00:00.400) 0:01:22.402 *********** 2021-07-26T05:58:22.442 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-07-26T05:58:22.444 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-07-26T05:58:22.465 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:22 +0000 (0:00:00.458) 0:01:22.861 *********** 2021-07-26T05:58:22.668 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-07-26T05:58:22.669 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-07-26T05:58:22.793 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-07-26T05:58:22.818 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:22 +0000 (0:00:00.352) 0:01:23.214 *********** 2021-07-26T05:58:22.885 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:22.908 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:22 +0000 (0:00:00.090) 0:01:23.304 *********** 2021-07-26T05:58:23.143 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi094.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-07-26T05:58:23.166 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:23 +0000 (0:00:00.257) 0:01:23.561 *********** 2021-07-26T05:58:23.369 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:23.393 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:23 +0000 (0:00:00.227) 0:01:23.789 *********** 2021-07-26T05:58:23.929 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-07-26T05:58:23.931 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-07-26T05:58:23.954 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:23 +0000 (0:00:00.561) 0:01:24.350 *********** 2021-07-26T05:58:24.223 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-07-26T05:58:24.224 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-07-26 05:56:51 UTC", "ActiveEnterTimestampMonotonic": "15357756", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-journald.socket network.target basic.target nss-lookup.target sysinit.target local-fs.target time-sync.target remote-fs.target system.slice systemd-tmpfiles-setup.service var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-07-26 05:56:51 UTC", "AssertTimestampMonotonic": "15355884", "Before": "xdm.service getty@tty1.service plymouth-quit.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-07-26 05:56:51 UTC", "Co 2021-07-26T05:58:24.226 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "15355884", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "896", "ExecMainStartTimestamp": "Mon 2021-07-26 05:56:51 UTC", "ExecMainStartTimestampMonotonic": "15357599", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Mon 2021-07-26 05:56:51 UTC] ; stop_time=[n/a] ; pid=896 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[Mon 2021-07-26 05:56:51 UTC] ; stop_time=[n/a] ; pid=896 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GI 2021-07-26T05:58:24.227 INFO:teuthology.task.ansible.out:D": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-07-26 05:56:51 UTC", "InactiveExitTimestampMonotonic": "15357756", "InvocationID": "1171370380c7468eadfae8c32ee35d13", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "896", "MemoryAccounting": "yes", "MemoryCurrent": "1384448", "MemoryD 2021-07-26T05:58:24.229 INFO:teuthology.task.ansible.out:enyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-07-26 05:56:51 UTC", "StateChangeTimestampMonotonic": "15357756", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no" 2021-07-26T05:58:24.231 INFO:teuthology.task.ansible.out:, "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38333", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-26T05:58:24.248 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:24 +0000 (0:00:00.293) 0:01:24.644 *********** 2021-07-26T05:58:24.309 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-07-26T05:58:24.311 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:24.333 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:24 +0000 (0:00:00.085) 0:01:24.729 *********** 2021-07-26T05:58:24.402 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-07-26T05:58:24.404 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:24.425 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:24 +0000 (0:00:00.091) 0:01:24.821 *********** 2021-07-26T05:58:24.494 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-07-26T05:58:24.496 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:24.518 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:24 +0000 (0:00:00.092) 0:01:24.914 *********** 2021-07-26T05:58:24.596 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-07-26T05:58:24.598 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:24.620 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:24 +0000 (0:00:00.102) 0:01:25.016 *********** 2021-07-26T05:58:24.847 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-07-26T05:58:24.849 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:24.871 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:24 +0000 (0:00:00.251) 0:01:25.267 *********** 2021-07-26T05:58:25.085 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:25.109 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:25 +0000 (0:00:00.238) 0:01:25.505 *********** 2021-07-26T05:58:25.312 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-07-26T05:58:25.314 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:25.335 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:25 +0000 (0:00:00.225) 0:01:25.731 *********** 2021-07-26T05:58:25.549 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:25.573 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:25 +0000 (0:00:00.238) 0:01:25.969 *********** 2021-07-26T05:58:25.765 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-07-26T05:58:25.767 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:25.907 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:25 +0000 (0:00:00.333) 0:01:26.303 *********** 2021-07-26T05:58:25.975 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-07-26T05:58:25.977 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:25.999 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:25 +0000 (0:00:00.091) 0:01:26.395 *********** 2021-07-26T05:58:26.066 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-07-26T05:58:26.068 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-07-26T05:58:26.090 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.091) 0:01:26.486 *********** 2021-07-26T05:58:26.168 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-07-26T05:58:26.170 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-07-26T05:58:26.193 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.103) 0:01:26.589 *********** 2021-07-26T05:58:26.283 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-07-26T05:58:26.285 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-07-26T05:58:26.308 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.114) 0:01:26.704 *********** 2021-07-26T05:58:26.386 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-07-26T05:58:26.389 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-07-26T05:58:26.411 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.102) 0:01:26.807 *********** 2021-07-26T05:58:26.541 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-07-26T05:58:26.543 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-07-26T05:58:26.565 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.154) 0:01:26.961 *********** 2021-07-26T05:58:26.747 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-07-26T05:58:26.749 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-07-26T05:58:26.772 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.206) 0:01:27.168 *********** 2021-07-26T05:58:26.860 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-07-26T05:58:26.862 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:26.884 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.112) 0:01:27.280 *********** 2021-07-26T05:58:26.953 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-07-26T05:58:26.955 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:26.976 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:26 +0000 (0:00:00.091) 0:01:27.372 *********** 2021-07-26T05:58:27.189 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi094.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-07-26T05:58:27.215 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:27 +0000 (0:00:00.238) 0:01:27.611 *********** 2021-07-26T05:58:27.654 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-07-26T05:58:27.656 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-07-26T05:58:27.678 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:27 +0000 (0:00:00.463) 0:01:28.074 *********** 2021-07-26T05:58:27.757 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-07-26T05:58:27.760 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi094"}, "changed": false} 2021-07-26T05:58:27.781 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:27 +0000 (0:00:00.102) 0:01:28.177 *********** 2021-07-26T05:58:28.532 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-07-26T05:58:28.533 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi094.front.sepia.ceph.com", "ansible_hostname": "smithi094", "ansible_nodename": "smithi094"}, "changed": true, "name": "smithi094"} 2021-07-26T05:58:28.556 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:28 +0000 (0:00:00.775) 0:01:28.952 *********** 2021-07-26T05:58:28.926 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-07-26T05:58:28.928 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-07-26T05:58:28.949 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:28 +0000 (0:00:00.393) 0:01:29.345 *********** 2021-07-26T05:58:29.328 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-07-26T05:58:29.330 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-07-26T05:58:29.351 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:29 +0000 (0:00:00.401) 0:01:29.747 *********** 2021-07-26T05:58:29.898 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-07-26T05:58:29.900 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-07-26T05:58:29.921 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:29 +0000 (0:00:00.569) 0:01:30.317 *********** 2021-07-26T05:58:29.989 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-07-26T05:58:29.990 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.012 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.091) 0:01:30.408 *********** 2021-07-26T05:58:30.080 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-07-26T05:58:30.082 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.104 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.091) 0:01:30.500 *********** 2021-07-26T05:58:30.172 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.196 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.092) 0:01:30.592 *********** 2021-07-26T05:58:30.264 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-26T05:58:30.266 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.289 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.092) 0:01:30.685 *********** 2021-07-26T05:58:30.358 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.419 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.092) 0:01:30.777 *********** 2021-07-26T05:58:30.448 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-26T05:58:30.450 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.473 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.092) 0:01:30.869 *********** 2021-07-26T05:58:30.544 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-26T05:58:30.545 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.569 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.095) 0:01:30.965 *********** 2021-07-26T05:58:30.639 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-07-26T05:58:30.640 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.663 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.093) 0:01:31.058 *********** 2021-07-26T05:58:30.743 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.766 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.103) 0:01:31.162 *********** 2021-07-26T05:58:30.854 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* skipping: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.881 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.114) 0:01:31.277 *********** 2021-07-26T05:58:30.950 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-07-26T05:58:30.952 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:30.978 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:30 +0000 (0:00:00.097) 0:01:31.374 *********** 2021-07-26T05:58:31.048 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-07-26T05:58:31.071 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.092) 0:01:31.467 *********** 2021-07-26T05:58:31.139 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-07-26T05:58:31.164 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.092) 0:01:31.559 *********** 2021-07-26T05:58:31.232 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-07-26T05:58:31.255 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.091) 0:01:31.651 *********** 2021-07-26T05:58:31.323 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-07-26T05:58:31.324 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:31.347 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.091) 0:01:31.743 *********** 2021-07-26T05:58:31.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-07-26T05:58:31.428 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-07-26T05:58:31.450 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.103) 0:01:31.846 *********** 2021-07-26T05:58:31.519 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-07-26T05:58:31.521 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:31.544 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.093) 0:01:31.940 *********** 2021-07-26T05:58:31.624 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:31.646 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.102) 0:01:32.042 *********** 2021-07-26T05:58:31.725 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:31.748 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.101) 0:01:32.144 *********** 2021-07-26T05:58:31.817 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:31.840 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.092) 0:01:32.236 *********** 2021-07-26T05:58:31.920 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-07-26T05:58:31.922 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:31.944 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:31 +0000 (0:00:00.103) 0:01:32.340 *********** 2021-07-26T05:58:32.013 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.037 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.093) 0:01:32.433 *********** 2021-07-26T05:58:32.107 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-07-26T05:58:32.109 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.135 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.097) 0:01:32.531 *********** 2021-07-26T05:58:32.204 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-26T05:58:32.206 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.227 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.092) 0:01:32.623 *********** 2021-07-26T05:58:32.307 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-26T05:58:32.309 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.330 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.102) 0:01:32.726 *********** 2021-07-26T05:58:32.398 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-07-26T05:58:32.401 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.422 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.092) 0:01:32.818 *********** 2021-07-26T05:58:32.501 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-26T05:58:32.503 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.525 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.102) 0:01:32.921 *********** 2021-07-26T05:58:32.593 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-26T05:58:32.595 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.617 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.091) 0:01:33.013 *********** 2021-07-26T05:58:32.695 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-07-26T05:58:32.697 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.718 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.101) 0:01:33.114 *********** 2021-07-26T05:58:32.787 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-07-26T05:58:32.789 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.810 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.091) 0:01:33.206 *********** 2021-07-26T05:58:32.889 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-07-26T05:58:32.891 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:32.913 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:32 +0000 (0:00:00.102) 0:01:33.309 *********** 2021-07-26T05:58:32.981 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-07-26T05:58:32.983 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:33.004 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:33 +0000 (0:00:00.091) 0:01:33.400 *********** 2021-07-26T05:58:33.083 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-07-26T05:58:33.084 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:33.112 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:33 +0000 (0:00:00.107) 0:01:33.508 *********** 2021-07-26T05:58:33.193 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-07-26T05:58:33.195 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:33.217 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:33 +0000 (0:00:00.104) 0:01:33.613 *********** 2021-07-26T05:58:33.296 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-07-26T05:58:33.298 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:33.324 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:33 +0000 (0:00:00.107) 0:01:33.720 *********** 2021-07-26T05:58:33.395 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-07-26T05:58:33.397 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:33.420 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:33 +0000 (0:00:00.095) 0:01:33.816 *********** 2021-07-26T05:58:33.602 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005387", "end": "2021-07-26 05:58:33.101390", "rc": 0, "start": "2021-07-26 05:58:33.096003", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T05:58:33.626 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:33 +0000 (0:00:00.205) 0:01:34.022 *********** 2021-07-26T05:58:33.694 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-07-26T05:58:33.719 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:33 +0000 (0:00:00.091) 0:01:34.113 *********** 2021-07-26T05:58:34.117 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-07-26T05:58:34.119 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-07-26T05:58:34.142 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:34 +0000 (0:00:00.424) 0:01:34.538 *********** 2021-07-26T05:58:34.210 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-07-26T05:58:34.212 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:58:34.234 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:34 +0000 (0:00:00.091) 0:01:34.630 *********** 2021-07-26T05:58:35.524 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-07-26T05:58:35.526 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-07-26T05:58:36.490 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-07-26T05:58:36.515 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:36 +0000 (0:00:02.281) 0:01:36.911 *********** 2021-07-26T05:58:37.250 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-07-26T05:58:37.252 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:58:37.275 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:37 +0000 (0:00:00.759) 0:01:37.671 *********** 2021-07-26T05:58:37.344 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-07-26T05:58:37.367 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:37 +0000 (0:00:00.092) 0:01:37.763 *********** 2021-07-26T05:58:37.585 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-07-26T05:58:37.587 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034760", "end": "2021-07-26 05:58:37.075040", "rc": 0, "start": "2021-07-26 05:58:37.040280", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T05:58:37.611 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:37 +0000 (0:00:00.243) 0:01:38.007 *********** 2021-07-26T05:58:39.791 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-07-26T05:58:39.793 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:58:39.816 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:39 +0000 (0:00:02.205) 0:01:40.212 *********** 2021-07-26T05:58:51.593 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-07-26T05:58:51.595 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 154 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138111 files and directories currently installed.)\r 2021-07-26T05:58:51.597 INFO:teuthology.task.ansible.out:\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-utils (1:4.2-3ubuntu6.14) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...\r\nRemoving librbd1 (15.2.8-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.8-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 154 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Re 2021-07-26T05:58:51.599 INFO:teuthology.task.ansible.out:ading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138111 files and directories currently installed.)", "Removing qemu-system-x86 (1:4.2-3ubuntu6.14) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.14) ...", "Removing qemu-utils (1:4.2-3ubuntu6.14) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.14) ...", "Removing librbd1 (15.2.8-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.8-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-07-26T05:58:51.619 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:58:51 +0000 (0:00:11.803) 0:01:52.015 *********** 2021-07-26T05:59:17.092 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-07-26T05:59:17.094 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apache2-bin apache2-data apache2-utils apt-utils libapt-pkg6.0\nSuggested packages:\n apache2-doc apache2-suexec-pristine | apache2-suexec-custom www-browser\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg6.0\n7 upgraded, 0 newly installed, 0 to remove and 147 not upgraded.\nNeed to get 3866 kB of archives.\nAfter this operation, 42.0 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libapt-pkg6.0 amd64 2.0.6 [835 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apt amd64 2.0.6 [1296 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apt-utils amd64 2.0.6 [216 kB]\n 2021-07-26T05:59:17.096 INFO:teuthology.task.ansible.out:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2 amd64 2.4.41-4ubuntu3.4 [95.5 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2-bin amd64 2.4.41-4ubuntu3.4 [1180 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2-data all 2.4.41-4ubuntu3.4 [159 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2-utils amd64 2.4.41-4ubuntu3.4 [84.0 kB]\nFetched 3866 kB in 1s (4069 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138039 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg6.0_2.0.6_amd64.deb ...\r\nUnpacking libapt-pkg6.0:amd64 (2.0.6) over (2.0.4) ...\r\nSetting up libapt-pkg6.0:amd64 (2.0.6) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138039 files and directories currently installed.)\r\nPreparing to unpack .../archives/apt_2.0.6_amd64.deb ...\r\nUnpacking apt (2.0.6 2021-07-26T05:59:17.097 INFO:teuthology.task.ansible.out:) over (2.0.4) ...\r\nSetting up apt (2.0.6) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138039 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_2.0.6_amd64.deb ...\r\nUnpacking apt-utils (2.0.6) over (2.0.4) ...\r\nPreparing to unpack .../apache2_2.4.41-4ubuntu3.4_amd64.deb ...\r\nUnpacking apache2 (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...\r\nPreparing to unpack .../apache2-bin_2.4.41-4ubuntu3.4_amd64.deb ...\r\nUnpacking apache2-bin (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...\r\nPreparing to unpack .../apache2-data_2.4.41-4ubuntu3.4_all.deb ...\r\nUnpacking apache2-data (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...\r\nPreparing to unpack .../apache2-utils_2.4.41-4ubuntu3.4_amd64.deb ...\r\nUnpacking apache2-utils (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...\r\nSetting up apt-utils (2.0.6) ...\r\nSetting up apache2-bin (2.4.41-4ubuntu3.4) ...\r\nSetting up apache2-data (2.4.41-4ubuntu3.4) ...\r\nSetting up apache2-utils (2.4.41-4ubuntu3.4) ...\r\nSetting up apache2 (2.4.41-4ubuntu3.4) ...\r\nProcessing triggers for ufw (0.36-6) ...\r\nProcessing triggers for systemd (245.4-4ubuntu3.4) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer re 2021-07-26T05:59:17.099 INFO:teuthology.task.ansible.out:quired:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apache2-bin apache2-data apache2-utils apt-utils libapt-pkg6.0", "Suggested packages:", " apache2-doc apache2-suexec-pristine | apache2-suexec-custom www-browser", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg6.0", "7 upgraded, 0 newly installed, 0 to remove and 147 not upgraded.", "Need to get 3866 kB of archives.", "After this operation, 42.0 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libapt-pkg6.0 amd64 2.0.6 [835 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apt amd64 2.0.6 [1296 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apt-utils amd64 2.0.6 [216 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2 amd64 2.4.41-4ubuntu3.4 [95.5 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2-bin amd64 2.4.41-4ubuntu3.4 [1180 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2-data all 2.4.41-4ubuntu3.4 [159 kB]", "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apache2-utils amd64 2.4.41-4ubuntu3.4 [84.0 kB] 2021-07-26T05:59:17.103 INFO:teuthology.task.ansible.out:", "Fetched 3866 kB in 1s (4069 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138039 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg6.0_2.0.6_amd64.deb ...", "Unpacking libapt-pkg6.0:amd64 (2.0.6) over (2.0.4) ...", "Setting up libapt-pkg6.0:amd64 (2.0.6) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138039 files and directories currently installed.)", "Preparing to unpack .../archives/apt_2.0.6_amd64.deb ...", "Unpacking apt (2.0.6) over (2.0.4) ...", "Setting up apt (2.0.6) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading dat 2021-07-26T05:59:17.105 INFO:teuthology.task.ansible.out:abase ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138039 files and directories currently installed.)", "Preparing to unpack .../apt-utils_2.0.6_amd64.deb ...", "Unpacking apt-utils (2.0.6) over (2.0.4) ...", "Preparing to unpack .../apache2_2.4.41-4ubuntu3.4_amd64.deb ...", "Unpacking apache2 (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...", "Preparing to unpack .../apache2-bin_2.4.41-4ubuntu3.4_amd64.deb ...", "Unpacking apache2-bin (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...", "Preparing to unpack .../apache2-data_2.4.41-4ubuntu3.4_all.deb ...", "Unpacking apache2-data (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...", "Preparing to unpack .../apache2-utils_2.4.41-4ubuntu3.4_amd64.deb ...", "Unpacking apache2-utils (2.4.41-4ubuntu3.4) over (2.4.41-4ubuntu3.1) ...", "Setting up apt-utils (2.0.6) ...", "Setting up apache2-bin (2.4.41-4ubuntu3.4) ...", "Setting up apache2-data (2.4.41-4ubuntu3.4) ...", "Setting up apache2-utils (2.4.41-4ubuntu3.4) ...", "Setting up apache2 (2.4.41-4ubuntu3.4) ...", "Processing triggers for ufw (0.36-6) ...", "Processing triggers for systemd (245.4-4ubuntu3.4) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-07-26T05:59:17.117 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:17 +0000 (0:00:25.498) 0:02:17.513 *********** 2021-07-26T05:59:17.188 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-07-26T05:59:17.190 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:17.212 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:17 +0000 (0:00:00.094) 0:02:17.608 *********** 2021-07-26T05:59:31.131 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-26T05:59:31.133 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-data\n qemu-system-gui qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git qemu-system-data qemu-system-gui\n3 upgraded, 6 newly installed, 0 to remove and 144 not upgraded.\nNeed to get 18.8 MB of archives.\nAfter this operation, 62.7 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.1 [4557 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.1 [3212 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.1 [1623 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.17 [1055 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.17 [563 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-gui amd64 1:4.2-3ubuntu6.17 [41.2 kB]\nGet:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x 2021-07-26T05:59:31.135 INFO:teuthology.task.ansible.out:86 amd64 1:4.2-3ubuntu6.17 [6738 kB]\nGet:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]\nFetched 18.8 MB in 1s (14.1 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138039 files and directories currently installed.)\r\nPreparing to unpack .../0-git_1%3a2.25.1-1ubuntu3.1_amd64.deb ...\r\nUnpacking git (1:2.25.1-1ubuntu3.1) over (1:2.25.1-1ubuntu3) ...\r\nSelecting previously unselected package librados2.\r\nPreparing to unpack .../1-librados2_15.2.13-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librados2 (15.2.13-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../2-librbd1_15.2.13-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librbd1 (15.2.13-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../3-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../4-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nPreparing to unpack .../5-qemu-system-data_1%3a4.2-3ubuntu6.17_all.deb ...\r\nUnpacking qemu-system-data (1:4.2-3ubuntu6.17) over (1:4.2-3ubuntu6.14) ...\r\nPreparing to unpack .../6-qemu-system-gui_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-gui:amd64 2021-07-26T05:59:31.136 INFO:teuthology.task.ansible.out: (1:4.2-3ubuntu6.17) over (1:4.2-3ubuntu6.14) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../7-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../8-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.17) ...\r\nSetting up librados2 (15.2.13-0ubuntu0.20.04.1) ...\r\nSetting up qemu-system-gui:amd64 (1:4.2-3ubuntu6.17) ...\r\nSetting up git (1:2.25.1-1ubuntu3.1) ...\r\nSetting up librbd1 (15.2.13-0ubuntu0.20.04.1) ...\r\nSetting up qemu-system-data (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.17) ...\r\nProcessing triggers for hicolor-icon-theme (0.17-2) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for mime-support (3.64ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-data", " qemu-system-gui qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git qemu-system-data qemu-system-gui", "3 upgraded, 6 newly installed, 0 to remove and 144 not upgraded.", "Need to get 18.8 MB of archives.", "After this operation, 62.7 MB of additional disk space will be used.", "Get:1 http://archive. 2021-07-26T05:59:31.138 INFO:teuthology.task.ansible.out:ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.1 [4557 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.1 [3212 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.1 [1623 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.17 [1055 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.17 [563 kB]", "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-gui amd64 1:4.2-3ubuntu6.17 [41.2 kB]", "Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.17 [6738 kB]", "Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]", "Fetched 18.8 MB in 1s (14.1 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138039 files and directories currently installed.)", "Preparing to unpack .../0-git_1%3a2.25.1-1ubuntu3.1_amd64.deb ...", "Unpacking git (1:2.25.1-1ubuntu3.1) over (1:2.25.1-1ubuntu3) ...", "Selecting previously unselected package librados2.", "Preparing to unpack .../1-librados2_15.2.13-0ubuntu0.20.04.1_amd64.deb . 2021-07-26T05:59:31.140 INFO:teuthology.task.ansible.out:..", "Unpacking librados2 (15.2.13-0ubuntu0.20.04.1) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../2-librbd1_15.2.13-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librbd1 (15.2.13-0ubuntu0.20.04.1) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../3-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../4-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.17) ...", "Preparing to unpack .../5-qemu-system-data_1%3a4.2-3ubuntu6.17_all.deb ...", "Unpacking qemu-system-data (1:4.2-3ubuntu6.17) over (1:4.2-3ubuntu6.14) ...", "Preparing to unpack .../6-qemu-system-gui_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-gui:amd64 (1:4.2-3ubuntu6.17) over (1:4.2-3ubuntu6.14) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../7-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../8-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.17) ...", "Setting up librados2 (15.2.13-0ubuntu0.20.04.1) ...", "Setting up qemu-system-gui:amd64 (1:4.2-3ubuntu6.17) ...", "Setting up git (1:2.25.1-1ubuntu3.1) ...", "Setting up librbd1 (15.2.13-0ubuntu0.20.04.1) ...", "Setting up qemu-system-data (1:4.2-3ubuntu6.17) ...", "Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.17) ...", "Setting 2021-07-26T05:59:31.140 INFO:teuthology.task.ansible.out: up qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Setting up qemu-utils (1:4.2-3ubuntu6.17) ...", "Processing triggers for hicolor-icon-theme (0.17-2) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing 2021-07-26T05:59:31.142 INFO:teuthology.task.ansible.out: triggers for mime-support (3.64ubuntu1) ..."]} 2021-07-26T05:59:31.165 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:31 +0000 (0:00:13.953) 0:02:31.561 *********** 2021-07-26T05:59:31.887 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-07-26T05:59:31.889 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:59:31.911 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:31 +0000 (0:00:00.745) 0:02:32.307 *********** 2021-07-26T05:59:32.643 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-07-26T05:59:32.646 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:59:32.678 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:32 +0000 (0:00:00.767) 0:02:33.074 *********** 2021-07-26T05:59:32.864 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-07-26T05:59:32.866 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-07-26T05:59:32.888 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:32 +0000 (0:00:00.209) 0:02:33.284 *********** 2021-07-26T05:59:32.981 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-07-26T05:59:32.982 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:33.007 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:33 +0000 (0:00:00.118) 0:02:33.402 *********** 2021-07-26T05:59:33.198 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-07-26T05:59:33.222 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:33 +0000 (0:00:00.215) 0:02:33.618 *********** 2021-07-26T05:59:33.670 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-07-26T05:59:33.693 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:33 +0000 (0:00:00.470) 0:02:34.088 *********** 2021-07-26T05:59:33.923 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-07-26T05:59:33.925 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-07-26T05:59:33.956 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:33 +0000 (0:00:00.263) 0:02:34.352 *********** 2021-07-26T05:59:34.141 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-07-26T05:59:34.143 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:59:34.175 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:34 +0000 (0:00:00.218) 0:02:34.571 *********** 2021-07-26T05:59:34.494 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-07-26T05:59:34.496 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2021-07-26 05:59:09 UTC", "ActiveEnterTimestampMonotonic": "153348574", "ActiveExitTimestamp": "Mon 2021-07-26 05:59:09 UTC", "ActiveExitTimestampMonotonic": "153256343", "ActiveState": "active", "After": "sysinit.target nss-lookup.target systemd-tmpfiles-setup.service system.slice network.target systemd-journald.socket -.mount remote-fs.target basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-07-26 05:59:09 UTC", "AssertTimestampMonotonic": "153308426", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-07-26 05:59:09 UTC", "ConditionTimestampMonotonic": "153308425", "ConfigurationDirec 2021-07-26T05:59:34.498 INFO:teuthology.task.ansible.out:toryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "6522", "ExecMainStartTimestamp": "Mon 2021-07-26 05:59:09 UTC", "ExecMainStartTimestampMonotonic": "153348543", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPrior 2021-07-26T05:59:34.500 INFO:teuthology.task.ansible.out:ity": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Mon 2021-07-26 05:59:09 UTC", "InactiveEnterTimestampMonotonic": "153307056", "InactiveExitTimestamp": "Mon 2021-07-26 05:59:09 UTC", "InactiveExitTimestampMonotonic": "153311301", "InvocationID": "71c2771b74124eec8b15a8845b3e644a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "6522", "MemoryAccounting": "yes", "MemoryCurrent": "6758400", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemoryS 2021-07-26T05:59:34.501 INFO:teuthology.task.ansible.out:wapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-07-26 05:59:09 UTC", "StateChangeTimestampMonotonic": "153348574", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", " 2021-07-26T05:59:34.503 INFO:teuthology.task.ansible.out:TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38333", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-26T05:59:34.537 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:34 +0000 (0:00:00.362) 0:02:34.933 *********** 2021-07-26T05:59:34.619 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-07-26T05:59:34.620 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:34.643 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:34 +0000 (0:00:00.105) 0:02:35.039 *********** 2021-07-26T05:59:34.725 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-07-26T05:59:34.726 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:34.748 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:34 +0000 (0:00:00.105) 0:02:35.144 *********** 2021-07-26T05:59:34.817 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-07-26T05:59:34.818 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:34.840 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:34 +0000 (0:00:00.091) 0:02:35.236 *********** 2021-07-26T05:59:34.914 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-07-26T05:59:34.916 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:34.950 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:34 +0000 (0:00:00.109) 0:02:35.346 *********** 2021-07-26T05:59:35.044 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-07-26T05:59:35.046 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.067 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.117) 0:02:35.463 *********** 2021-07-26T05:59:35.148 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-07-26T05:59:35.150 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.179 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.111) 0:02:35.575 *********** 2021-07-26T05:59:35.251 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.280 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.100) 0:02:35.676 *********** 2021-07-26T05:59:35.363 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-07-26T05:59:35.365 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.386 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.106) 0:02:35.782 *********** 2021-07-26T05:59:35.467 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.490 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.104) 0:02:35.886 *********** 2021-07-26T05:59:35.570 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-07-26T05:59:35.572 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.596 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.105) 0:02:35.992 *********** 2021-07-26T05:59:35.667 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-07-26T05:59:35.668 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.693 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.097) 0:02:36.089 *********** 2021-07-26T05:59:35.762 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.790 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.097) 0:02:36.186 *********** 2021-07-26T05:59:35.863 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-26T05:59:35.865 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.886 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.096) 0:02:36.282 *********** 2021-07-26T05:59:35.955 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:35.980 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:35 +0000 (0:00:00.093) 0:02:36.376 *********** 2021-07-26T05:59:36.049 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-26T05:59:36.051 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.073 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.093) 0:02:36.469 *********** 2021-07-26T05:59:36.143 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-26T05:59:36.145 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.167 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.093) 0:02:36.563 *********** 2021-07-26T05:59:36.238 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.261 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.094) 0:02:36.657 *********** 2021-07-26T05:59:36.330 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-26T05:59:36.332 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.356 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.095) 0:02:36.752 *********** 2021-07-26T05:59:36.436 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-07-26T05:59:36.437 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.459 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.102) 0:02:36.855 *********** 2021-07-26T05:59:36.528 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-26T05:59:36.529 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.551 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.091) 0:02:36.947 *********** 2021-07-26T05:59:36.630 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.656 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.104) 0:02:37.051 *********** 2021-07-26T05:59:36.725 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-26T05:59:36.727 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.750 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.094) 0:02:37.146 *********** 2021-07-26T05:59:36.819 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-07-26T05:59:36.821 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:36.842 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:36 +0000 (0:00:00.092) 0:02:37.238 *********** 2021-07-26T05:59:36.912 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-07-26T05:59:37.009 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 26 July 2021 05:59:36 +0000 (0:00:00.101) 0:02:37.340 *********** 2021-07-26T05:59:37.024 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-07-26T05:59:37.026 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:37.048 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:37 +0000 (0:00:00.104) 0:02:37.444 *********** 2021-07-26T05:59:37.118 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-07-26T05:59:37.120 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:37.152 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:37 +0000 (0:00:00.103) 0:02:37.548 *********** 2021-07-26T05:59:37.339 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-07-26T05:59:37.341 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-07-26T05:59:37.364 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:37 +0000 (0:00:00.211) 0:02:37.760 *********** 2021-07-26T05:59:37.759 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-07-26T05:59:37.761 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-07-26T05:59:37.782 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:37 +0000 (0:00:00.418) 0:02:38.178 *********** 2021-07-26T05:59:37.854 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-07-26T05:59:37.856 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:37.878 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:37 +0000 (0:00:00.095) 0:02:38.274 *********** 2021-07-26T05:59:38.290 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-07-26T05:59:38.312 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:38 +0000 (0:00:00.433) 0:02:38.708 *********** 2021-07-26T05:59:38.692 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-07-26T05:59:38.715 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:38 +0000 (0:00:00.403) 0:02:39.111 *********** 2021-07-26T05:59:39.106 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-07-26T05:59:39.132 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.416) 0:02:39.528 *********** 2021-07-26T05:59:39.317 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:59:39.341 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.209) 0:02:39.737 *********** 2021-07-26T05:59:39.413 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:39.438 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.095) 0:02:39.833 *********** 2021-07-26T05:59:39.507 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-07-26T05:59:39.509 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:39.531 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.094) 0:02:39.927 *********** 2021-07-26T05:59:39.600 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:39.624 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.092) 0:02:40.020 *********** 2021-07-26T05:59:39.703 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-07-26T05:59:39.705 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:39.728 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.103) 0:02:40.124 *********** 2021-07-26T05:59:39.797 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:39.820 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.092) 0:02:40.216 *********** 2021-07-26T05:59:39.906 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T05:59:39.908 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-07-26T05:59:39.930 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:39 +0000 (0:00:00.109) 0:02:40.326 *********** 2021-07-26T05:59:40.021 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-07-26T05:59:40.023 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.032 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.034 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.044 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.045 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.055 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-07-26T05:59:40.066 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.068 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.078 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.080 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.090 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.113 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.115 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.143 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:40 +0000 (0:00:00.213) 0:02:40.539 *********** 2021-07-26T05:59:40.394 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-07-26T05:59:40.396 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003596", "end": "2021-07-26 05:59:40.368202", "rc": 0, "start": "2021-07-26 05:59:40.364606", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/bin/teuthology-lock"]} 2021-07-26T05:59:40.420 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:40 +0000 (0:00:00.277) 0:02:40.816 *********** 2021-07-26T05:59:40.501 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.531 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:40 +0000 (0:00:00.110) 0:02:40.927 *********** 2021-07-26T05:59:40.611 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-07-26T05:59:40.613 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.639 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:40 +0000 (0:00:00.108) 0:02:41.035 *********** 2021-07-26T05:59:40.709 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-07-26T05:59:40.711 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:40.733 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:40 +0000 (0:00:00.094) 0:02:41.129 *********** 2021-07-26T05:59:41.442 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-07-26T05:59:41.444 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T05:59:41.466 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:41 +0000 (0:00:00.732) 0:02:41.862 *********** 2021-07-26T05:59:41.546 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:41.570 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:41 +0000 (0:00:00.103) 0:02:41.966 *********** 2021-07-26T05:59:41.653 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-26T05:59:41.654 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item={'block_used': 13859352, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226235756, 'size_available': 926661656576, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150120, 'block_size': 4096, 'inode_available': 60904856}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226235756, "block_size": 4096, "block_total": 240095108, "block_used": 13859352, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904856, "inode_total": 61054976, "inode_used": 150120, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926661656576, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-07-26T05:59:41.678 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:41 +0000 (0:00:00.108) 0:02:42.074 *********** 2021-07-26T05:59:41.876 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-07-26T05:59:41.878 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009287", "end": "2021-07-26 05:59:41.370657", "rc": 0, "start": "2021-07-26 05:59:41.361370", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:41.900 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:41 +0000 (0:00:00.222) 0:02:42.296 *********** 2021-07-26T05:59:41.991 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-07-26T05:59:41.994 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'block_used': 13859352, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226235756, 'size_available': 926661656576, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150120, 'block_size': 4096, 'inode_available': 60904856}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226235756, "block_size": 4096, "block_total": 240095108, "block_used": 13859352, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904856, "inode_total": 61054976, "inode_used": 150120, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926661656576, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:42.016 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:42 +0000 (0:00:00.115) 0:02:42.412 *********** 2021-07-26T05:59:42.084 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-07-26T05:59:42.108 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:42 +0000 (0:00:00.091) 0:02:42.504 *********** 2021-07-26T05:59:42.208 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-07-26T05:59:42.209 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.951 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASM2ZqOFCo7JzCJR8hTCW7Q2uFIFpFAodnj'], 'uuids': ['d964a5ab-f29b-4d9f-beaa-fc2b08cb4672']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.612757", "end": "2021-07-26 05:59:43.437428", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASM2ZqOFCo7JzCJR8hTCW7Q2uFIFpFAodnj"], "labels": [], "masters": [], "uuids": ["d964a5ab-f29b-4d9f-beaa-fc2b08cb4672"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:41.824671", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T05:59:43.963 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:43.964 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:45.225 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMP0TNxf95fybvwejFVDTqc4cBlGVTqQ4N'], 'uuids': ['9bfcba70-2843-41d3-b938-6b10dd78c5f4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.128457", "end": "2021-07-26 05:59:44.708079", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMP0TNxf95fybvwejFVDTqc4cBlGVTqQ4N"], "labels": [], "masters": [], "uuids": ["9bfcba70-2843-41d3-b938-6b10dd78c5f4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:43.579622", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T05:59:45.227 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP', 'wwn-0x5000c500918dbfac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'wwn-0x5000c500918dbfac-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918dbfac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJNP', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP", "wwn-0x5000c500918dbfac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1", "wwn-0x5000c500918dbfac-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJNP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dbfac"}}, 2021-07-26T05:59:45.229 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-07-26T05:59:46.496 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-FZKHpV-B3TT-92Bo-ohux-qjJy-q2Pu-ZtrJN8', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN', 'nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.135984", "end": "2021-07-26 05:59:45.984456", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-FZKHpV-B3TT-92Bo-ohux-qjJy-q2Pu-ZtrJN8", "nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN", "nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:44.848472", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures destroyed! You 2021-07-26T05:59:46.498 INFO:teuthology.task.ansible.out:may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T05:59:46.508 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:46.520 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:47.775 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMzM1cMy3miGum7hPKXkfGycacg6kbAE0S'], 'uuids': ['1607a260-5ea0-4916-86d0-de79bfa7911e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.132285", "end": "2021-07-26 05:59:47.262512", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMzM1cMy3miGum7hPKXkfGycacg6kbAE0S"], "labels": [], "masters": [], "uuids": ["1607a260-5ea0-4916-86d0-de79bfa7911e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:46.130227", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T05:59:47.785 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:47.787 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:49.068 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMGXeurkEQmvdTp3oozTMrnOpTb7fBXFps'], 'uuids': ['4eace86e-847e-4993-a72e-cfba41b71bd8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.136147", "end": "2021-07-26 05:59:48.546006", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMGXeurkEQmvdTp3oozTMrnOpTb7fBXFps"], "labels": [], "masters": [], "uuids": ["4eace86e-847e-4993-a72e-cfba41b71bd8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:47.409859", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T05:59:49.070 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:50.331 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMaA961CJqRqDSGPh6AdgDNKSdUMmNVvby'], 'uuids': ['13a1f372-0d82-492e-8ceb-1960a12f2809']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.129572", "end": "2021-07-26 05:59:49.812975", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMaA961CJqRqDSGPh6AdgDNKSdUMmNVvby"], "labels": [], "masters": [], "uuids": ["13a1f372-0d82-492e-8ceb-1960a12f2809"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:48.683403", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-07-26T05:59:50.369 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:50 +0000 (0:00:08.259) 0:02:50.763 *********** 2021-07-26T05:59:50.457 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-07-26T05:59:50.458 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:50.697 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASM2ZqOFCo7JzCJR8hTCW7Q2uFIFpFAodnj'], 'uuids': ['d964a5ab-f29b-4d9f-beaa-fc2b08cb4672']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.112182", "end": "2021-07-26 05:59:50.186997", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASM2ZqOFCo7JzCJR8hTCW7Q2uFIFpFAodnj"], "labels": [], "masters": [], "uuids": ["d964a5ab-f29b-4d9f-beaa-fc2b08cb4672"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:50.074815", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.108964 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.108964 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:50.709 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:50.720 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:50.961 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMP0TNxf95fybvwejFVDTqc4cBlGVTqQ4N'], 'uuids': ['9bfcba70-2843-41d3-b938-6b10dd78c5f4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.120104", "end": "2021-07-26 05:59:50.452723", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMP0TNxf95fybvwejFVDTqc4cBlGVTqQ4N"], "labels": [], "masters": [], "uuids": ["9bfcba70-2843-41d3-b938-6b10dd78c5f4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:50.332619", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.115172 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.115172 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:50.973 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP', 'wwn-0x5000c500918dbfac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1', 'wwn-0x5000c500918dbfac-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918dbfac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJNP', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP", "wwn-0x5000c500918dbfac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJNP-part1", "wwn-0x5000c500918dbfac-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJNP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918dbfac"}}, 2021-07-26T05:59:50.975 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-07-26T05:59:51.163 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-FZKHpV-B3TT-92Bo-ohux-qjJy-q2Pu-ZtrJN8', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN', 'nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054121", "end": "2021-07-26 05:59:50.641154", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-FZKHpV-B3TT-92Bo-ohux-qjJy-q2Pu-ZtrJN8", "nvme-INTEL_SSDPED1D480GA_PHMB7513001L480DGN", "nvme-nvme.8086-50484d42373531333030314c34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:50.587033", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0494187 s, 2.3 GB/s", "stderr_lines": [" 2021-07-26T05:59:51.165 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0494187 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:51.176 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:51.410 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMzM1cMy3miGum7hPKXkfGycacg6kbAE0S'], 'uuids': ['1607a260-5ea0-4916-86d0-de79bfa7911e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.114844", "end": "2021-07-26 05:59:50.896535", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMzM1cMy3miGum7hPKXkfGycacg6kbAE0S"], "labels": [], "masters": [], "uuids": ["1607a260-5ea0-4916-86d0-de79bfa7911e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:50.781691", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.110447 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.110447 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:51.421 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:51.422 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:51.655 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMGXeurkEQmvdTp3oozTMrnOpTb7fBXFps'], 'uuids': ['4eace86e-847e-4993-a72e-cfba41b71bd8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.115581", "end": "2021-07-26 05:59:51.150578", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMGXeurkEQmvdTp3oozTMrnOpTb7fBXFps"], "labels": [], "masters": [], "uuids": ["4eace86e-847e-4993-a72e-cfba41b71bd8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:51.034997", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.110943 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.110943 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:51.667 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:51.910 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMaA961CJqRqDSGPh6AdgDNKSdUMmNVvby'], 'uuids': ['13a1f372-0d82-492e-8ceb-1960a12f2809']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.115268", "end": "2021-07-26 05:59:51.396136", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-EOPlu09DftbsBmSHuqDkk3zXG7xQ1ASMaA961CJqRqDSGPh6AdgDNKSdUMmNVvby"], "labels": [], "masters": [], "uuids": ["13a1f372-0d82-492e-8ceb-1960a12f2809"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-07-26 05:59:51.280868", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.110617 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.110617 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:51.944 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:51 +0000 (0:00:01.576) 0:02:52.340 *********** 2021-07-26T05:59:52.276 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi094.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.128755", "end": "2021-07-26 05:59:51.700721", "rc": 0, "start": "2021-07-26 05:59:51.571966", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:52.311 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:52 +0000 (0:00:00.356) 0:02:52.696 *********** 2021-07-26T05:59:52.516 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.071318", "end": "2021-07-26 05:59:52.001341", "rc": 0, "start": "2021-07-26 05:59:51.930023", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T05:59:52.542 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:52 +0000 (0:00:00.240) 0:02:52.937 *********** 2021-07-26T05:59:52.611 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-07-26T05:59:52.674 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:52 +0000 (0:00:00.132) 0:02:53.070 *********** 2021-07-26T05:59:52.701 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-07-26T05:59:52.703 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:52.727 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:52 +0000 (0:00:00.052) 0:02:53.122 *********** 2021-07-26T05:59:52.753 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-07-26T05:59:52.756 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:52.777 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:52 +0000 (0:00:00.050) 0:02:53.172 *********** 2021-07-26T05:59:52.803 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-26T05:59:52.805 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:52.829 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:52 +0000 (0:00:00.052) 0:02:53.225 *********** 2021-07-26T05:59:52.856 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-07-26T05:59:52.858 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:52.879 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:52 +0000 (0:00:00.050) 0:02:53.275 *********** 2021-07-26T05:59:53.012 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-07-26T05:59:53.038 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.158) 0:02:53.433 *********** 2021-07-26T05:59:53.086 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-07-26T05:59:53.088 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'block_used': 13859352, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226235756, 'size_available': 926661656576, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150120, 'block_size': 4096, 'inode_available': 60904856}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226235756, "block_size": 4096, "block_total": 240095108, "block_used": 13859352, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904856, "inode_total": 61054976, "inode_used": 150120, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926661656576, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:53.112 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.072) 0:02:53.506 *********** 2021-07-26T05:59:53.152 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:53.175 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.064) 0:02:53.571 *********** 2021-07-26T05:59:53.212 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T05:59:53.214 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:53.236 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.060) 0:02:53.631 *********** 2021-07-26T05:59:53.273 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T05:59:53.275 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:53.296 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.060) 0:02:53.692 *********** 2021-07-26T05:59:53.323 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:53.347 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.051) 0:02:53.743 *********** 2021-07-26T05:59:53.374 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:53.399 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.051) 0:02:53.794 *********** 2021-07-26T05:59:53.439 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-07-26T05:59:53.443 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:53.464 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:53 +0000 (0:00:00.065) 0:02:53.860 *********** 2021-07-26T05:59:54.275 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-07-26T05:59:54.277 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-07-26T05:59:54.299 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:54 +0000 (0:00:00.835) 0:02:54.695 *********** 2021-07-26T05:59:55.067 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-07-26T05:59:55.069 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T05:59:55.435 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T05:59:55.860 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T05:59:56.258 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T05:59:56.647 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-07-26T05:59:56.673 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:02.373) 0:02:57.069 *********** 2021-07-26T05:59:56.818 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-07-26T05:59:56.820 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-07-26T05:59:56.843 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:56 +0000 (0:00:00.169) 0:02:57.238 *********** 2021-07-26T05:59:56.996 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-07-26T05:59:56.998 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T05:59:57.134 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T05:59:57.254 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T05:59:57.382 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-07-26T05:59:57.393 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.419 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.576) 0:02:57.815 *********** 2021-07-26T05:59:57.446 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-07-26T05:59:57.448 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.469 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.050) 0:02:57.865 *********** 2021-07-26T05:59:57.496 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-07-26T05:59:57.498 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:57.522 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.052) 0:02:57.918 *********** 2021-07-26T05:59:57.665 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-07-26T05:59:57.666 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T05:59:57.689 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.166) 0:02:58.085 *********** 2021-07-26T05:59:57.726 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-07-26T05:59:57.727 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-07-26T05:59:57.752 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:57 +0000 (0:00:00.063) 0:02:58.148 *********** 2021-07-26T05:59:58.235 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi094.front.sepia.ceph.com] => {"changed": true} 2021-07-26T05:59:58.260 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.508) 0:02:58.656 *********** 2021-07-26T05:59:58.779 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-07-26T05:59:58.781 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-07-26T05:59:58.805 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.544) 0:02:59.200 *********** 2021-07-26T05:59:58.832 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-07-26T05:59:58.834 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:58.856 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.050) 0:02:59.251 *********** 2021-07-26T05:59:58.882 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-07-26T05:59:58.884 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:58.907 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.051) 0:02:59.303 *********** 2021-07-26T05:59:58.934 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-26T05:59:58.935 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:58.957 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:58 +0000 (0:00:00.049) 0:02:59.352 *********** 2021-07-26T05:59:58.983 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-07-26T05:59:58.985 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T05:59:59.007 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:59 +0000 (0:00:00.050) 0:02:59.403 *********** 2021-07-26T05:59:59.172 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-07-26T05:59:59.174 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-07-26T05:59:59.197 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:59 +0000 (0:00:00.189) 0:02:59.593 *********** 2021-07-26T05:59:59.350 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-07-26T05:59:59.351 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T05:59:59.377 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:59 +0000 (0:00:00.179) 0:02:59.772 *********** 2021-07-26T05:59:59.723 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-07-26T05:59:59.725 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-07-26T05:59:59.747 INFO:teuthology.task.ansible.out:Monday 26 July 2021 05:59:59 +0000 (0:00:00.370) 0:03:00.143 *********** 2021-07-26T06:00:00.092 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-07-26T06:00:00.093 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-07-26T06:00:00.121 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:00 +0000 (0:00:00.374) 0:03:00.517 *********** 2021-07-26T06:00:00.790 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-07-26T06:00:00.792 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T06:00:00.815 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:00 +0000 (0:00:00.693) 0:03:01.211 *********** 2021-07-26T06:00:01.095 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-07-26T06:00:01.097 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.132664", "end": "2021-07-26 06:00:00.587250", "rc": 0, "start": "2021-07-26 06:00:00.454586", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} 2021-07-26T06:00:01.119 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.303) 0:03:01.515 *********** 2021-07-26T06:00:01.146 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-07-26T06:00:01.147 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.170 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.050) 0:03:01.566 *********** 2021-07-26T06:00:01.196 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.222 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.052) 0:03:01.618 *********** 2021-07-26T06:00:01.893 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-07-26T06:00:01.895 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T06:00:01.917 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.694) 0:03:02.313 *********** 2021-07-26T06:00:01.944 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-07-26T06:00:01.945 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:01.968 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:01 +0000 (0:00:00.050) 0:03:02.364 *********** 2021-07-26T06:00:02.310 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-07-26T06:00:02.312 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-07-26T06:00:02.338 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:02 +0000 (0:00:00.366) 0:03:02.730 *********** 2021-07-26T06:00:02.361 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-07-26T06:00:02.363 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:02.386 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:02 +0000 (0:00:00.051) 0:03:02.782 *********** 2021-07-26T06:00:02.626 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-07-26 05:56:49 UTC", "ActiveEnterTimestampMonotonic": "13478875", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service system.slice sysinit.target systemd-journald.socket network.target basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-07-26 05:56:49 UTC", "AssertTimestampMonotonic": "13171724", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-07-26 05:56:49 UTC", "ConditionTimestampMonotonic": "13171723", "ConfigurationDirectoryMode 2021-07-26T06:00:02.628 INFO:teuthology.task.ansible.out:": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "834", "ExecMainStartTimestamp": "Mon 2021-07-26 05:56:49 UTC", "ExecMainStartTimestampMonotonic": "13478861", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-07-26 05:56:49 UTC", "InactiveExitTimestampMonotonic": "13172581", "InvocationID": "436ddc1cc5f7449fa8aaec9fade90888", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": " 2021-07-26T06:00:02.630 INFO:teuthology.task.ansible.out:15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "834", "MemoryAccounting": "yes", "MemoryCurrent": "3514368", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit" 2021-07-26T06:00:02.631 INFO:teuthology.task.ansible.out:: "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-07-26 05:56:49 UTC", "StateChangeTimestampMonotonic": "13478875", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38333", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-26T06:00:02.651 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:02 +0000 (0:00:00.265) 0:03:03.047 *********** 2021-07-26T06:00:02.690 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:00:02.692 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-07-26T06:00:02.714 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:02 +0000 (0:00:00.062) 0:03:03.110 *********** 2021-07-26T06:00:02.752 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:00:02.754 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-07-26T06:00:02.777 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:02 +0000 (0:00:00.063) 0:03:03.173 *********** 2021-07-26T06:00:02.803 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-07-26T06:00:02.805 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:02.828 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:02 +0000 (0:00:00.051) 0:03:03.224 *********** 2021-07-26T06:00:03.940 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-07-26T06:00:03.941 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T06:00:03.963 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:03 +0000 (0:00:01.135) 0:03:04.359 *********** 2021-07-26T06:00:03.990 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-07-26T06:00:03.992 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:04.014 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:04 +0000 (0:00:00.050) 0:03:04.410 *********** 2021-07-26T06:00:04.192 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-07-26T06:00:04.216 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:04 +0000 (0:00:00.201) 0:03:04.611 *********** 2021-07-26T06:00:04.660 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-07-26T06:00:04.662 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-07-26T06:00:04.684 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:04 +0000 (0:00:00.468) 0:03:05.080 *********** 2021-07-26T06:00:07.058 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-07-26T06:00:07.060 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-07-26T06:00:07.083 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:02.399) 0:03:07.479 *********** 2021-07-26T06:00:07.603 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-07-26T06:00:07.605 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false} 2021-07-26T06:00:07.636 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:00.553) 0:03:08.032 *********** 2021-07-26T06:00:07.690 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} Monday 26 July 2021 06:00:07 +0000 (0:00:00.050) 0:03:08.083 *********** 2021-07-26T06:00:07.872 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-07-26T06:00:07.873 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1627278838.4098475, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1627278838.4058475, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1627278838.4058475, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-07-26T06:00:07.896 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:07 +0000 (0:00:00.208) 0:03:08.292 *********** 2021-07-26T06:00:08.362 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-07-26T06:00:08.364 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-07-26T06:00:08.387 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:08 +0000 (0:00:00.491) 0:03:08.783 *********** 2021-07-26T06:00:08.414 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-07-26T06:00:08.416 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-07-26T06:00:08.438 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:08 +0000 (0:00:00.050) 0:03:08.834 *********** 2021-07-26T06:00:08.464 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-07-26T06:00:08.466 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:08.488 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:08 +0000 (0:00:00.050) 0:03:08.884 *********** 2021-07-26T06:00:08.672 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-07-26T06:00:08.674 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-07-26T06:00:08.696 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:08 +0000 (0:00:00.208) 0:03:09.092 *********** 2021-07-26T06:00:08.862 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-07-26T06:00:08.864 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-07-26T06:00:08.886 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:08 +0000 (0:00:00.189) 0:03:09.282 *********** 2021-07-26T06:00:08.913 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-07-26T06:00:08.915 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:08.936 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:08 +0000 (0:00:00.050) 0:03:09.332 *********** 2021-07-26T06:00:08.974 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-07-26T06:00:08.976 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:08.985 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:08.987 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi094.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.013 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.076) 0:03:09.408 *********** 2021-07-26T06:00:09.050 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-07-26T06:00:09.052 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.061 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.063 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.073 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.075 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.099 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.086) 0:03:09.495 *********** 2021-07-26T06:00:09.127 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-07-26T06:00:09.128 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.156 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.056) 0:03:09.552 *********** 2021-07-26T06:00:09.183 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-07-26T06:00:09.185 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.210 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.053) 0:03:09.605 *********** 2021-07-26T06:00:09.237 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-07-26T06:00:09.239 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.268 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.058) 0:03:09.664 *********** 2021-07-26T06:00:09.295 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-07-26T06:00:09.296 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.327 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.059) 0:03:09.723 *********** 2021-07-26T06:00:09.354 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-07-26T06:00:09.356 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.380 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.052) 0:03:09.776 *********** 2021-07-26T06:00:09.407 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.433 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.053) 0:03:09.829 *********** 2021-07-26T06:00:09.461 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-07-26T06:00:09.463 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.487 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.053) 0:03:09.883 *********** 2021-07-26T06:00:09.514 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:09.538 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:09 +0000 (0:00:00.051) 0:03:09.934 *********** 2021-07-26T06:00:09.743 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-07-26T06:00:09.745 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010428", "end": "2021-07-26 06:00:09.714854", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-07-26 06:00:09.704426", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:00:09.903 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006024", "end": "2021-07-26 06:00:09.874936", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-07-26 06:00:09.868912", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:00:10.060 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005511", "end": "2021-07-26 06:00:10.030758", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-07-26 06:00:10.025247", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-07-26T06:00:10.247 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.030495", "end": "2021-07-26 06:00:10.215168", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-07-26 06:00:10.184673", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:00:10.273 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:10 +0000 (0:00:00.734) 0:03:10.668 *********** 2021-07-26T06:00:10.300 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-07-26T06:00:10.323 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:10 +0000 (0:00:00.050) 0:03:10.719 *********** 2021-07-26T06:00:10.350 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-07-26T06:00:10.352 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:10.421 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-07-26T06:00:10.443 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:10 +0000 (0:00:00.120) 0:03:10.839 *********** 2021-07-26T06:00:11.434 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi094.front.sepia.ceph.com] 2021-07-26T06:00:11.488 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:11 +0000 (0:00:01.045) 0:03:11.884 *********** 2021-07-26T06:00:11.516 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-07-26T06:00:11.540 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:11 +0000 (0:00:00.052) 0:03:11.936 *********** 2021-07-26T06:00:11.567 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-26T06:00:11.569 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:11.590 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:11 +0000 (0:00:00.049) 0:03:11.986 *********** 2021-07-26T06:00:11.627 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi094.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-07-26T06:00:11.652 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:11 +0000 (0:00:00.062) 0:03:12.048 *********** 2021-07-26T06:00:27.879 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-07-26T06:00:27.881 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n aufs-tools btrfs-progs debootstrap docker-doc rinse zfs-fuse | zfsutils\nThe following packages will be upgraded:\n docker.io\n1 upgraded, 0 newly installed, 0 to remove and 143 not upgraded.\nNeed to get 36.9 MB of archives.\nAfter this operation, 931 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.2-0ubuntu1~20.04.2 [36.9 MB]\nPreconfiguring packages ...\nFetched 36.9 MB in 2s (18.8 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138111 files and directories currently installed.)\r\nPreparing to unpack .../docker.io_20.10.2-0ubuntu1~20.04.2_amd64.deb ...\r\nUnpacking docker.io (20.10.2-0ubuntu1~20.04.2) over (19.03.8-0ubuntu1.20.04.2) ...\r\nSetting up docker.io (20.10.2-0ubuntu1~20.04.2) ...\r\nCreated symlink /etc/systemd/system/multi-user.target.wants/docker.service -> /lib/systemd/system/docker.service.\r\nProcessing triggers for man-db (2.9.1-1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Suggested packages:", " aufs-tools btrfs-progs debootstrap docker-doc rinse zfs-fuse | zfsutils", "The followin 2021-07-26T06:00:27.883 INFO:teuthology.task.ansible.out:g packages will be upgraded:", " docker.io", "1 upgraded, 0 newly installed, 0 to remove and 143 not upgraded.", "Need to get 36.9 MB of archives.", "After this operation, 931 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.2-0ubuntu1~20.04.2 [36.9 MB]", "Preconfiguring packages ...", "Fetched 36.9 MB in 2s (18.8 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138111 files and directories currently installed.)", "Preparing to unpack .../docker.io_20.10.2-0ubuntu1~20.04.2_amd64.deb ...", "Unpacking docker.io (20.10.2-0ubuntu1~20.04.2) over (19.03.8-0ubuntu1.20.04.2) ...", "Setting up docker.io (20.10.2-0ubuntu1~20.04.2) ...", "Created symlink /etc/systemd/system/multi-user.target.wants/docker.service -> /lib/systemd/system/docker.service.", "Processing triggers for man-db (2.9.1-1) ..."]} 2021-07-26T06:00:27.907 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:27 +0000 (0:00:16.251) 0:03:28.299 *********** 2021-07-26T06:00:27.930 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-26T06:00:27.932 INFO:teuthology.task.ansible.out:skipping: [smithi094.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-07-26T06:00:27.954 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:27 +0000 (0:00:00.050) 0:03:28.350 *********** 2021-07-26T06:00:27.991 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-07-26T06:00:27.993 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-07-26T06:00:28.016 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:28 +0000 (0:00:00.061) 0:03:28.411 *********** 2021-07-26T06:00:28.213 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-07-26T06:00:28.215 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T06:00:28.371 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-07-26T06:00:28.397 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:28 +0000 (0:00:00.381) 0:03:28.793 *********** 2021-07-26T06:00:28.854 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-07-26T06:00:28.856 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279228.4545195-18080-227790677929711/source", "state": "file", "uid": 0} 2021-07-26T06:00:29.279 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279228.8815312-18080-237690129657592/source", "state": "file", "uid": 0} 2021-07-26T06:00:29.304 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:29 +0000 (0:00:00.906) 0:03:29.700 *********** 2021-07-26T06:00:30.066 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** ok: [smithi094.front.sepia.ceph.com] => {"cache_update_time": 1627278855, "cache_updated": false, "changed": false} 2021-07-26T06:00:30.102 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:30 +0000 (0:00:00.798) 0:03:30.498 *********** 2021-07-26T06:00:35.055 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-07-26T06:00:35.057 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-ixmzo1we\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-ixmzo1we"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-ixmzo1we\nCollecting docopt\n Downloading docopt-0.6.2.tar.gz (25 kB)\nCollecting toml\n Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)\nBuilding wheels for collected packages: registries-conf-ctl, docopt\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=3ff4bf7d78245fc36ece7d3625db13077223f7c06cca38826397a9d003c06365\n Stored in directory: /tmp/pip-ephem-wheel-cache-103kykro/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\n Building wheel for docopt (setup.py): started\n Building wheel for docopt (setup.py): finished with status 'done'\n Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=54209e9345d79075d7ce0aaf7039c649b8e425254909967f2661d2b34965736e\n Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c\nSuccessfully built registries-conf-ctl docopt\nInstalling collected packages: docopt, toml, registries-conf-ctl\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github. 2021-07-26T06:00:35.059 INFO:teuthology.task.ansible.out:com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-ixmzo1we", "Collecting docopt", " Downloading docopt-0.6.2.tar.gz (25 kB)", "Collecting toml", " Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)", "Building wheels for collected packages: registries-conf-ctl, docopt", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=6035 sha256=3ff4bf7d78245fc36ece7d3625db13077223f7c06cca38826397a9d003c06365", " Stored in directory: /tmp/pip-ephem-wheel-cache-103kykro/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", " Building wheel for docopt (setup.py): started", " Building wheel for docopt (setup.py): finished with status 'done'", " Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=54209e9345d79075d7ce0aaf7039c649b8e425254909967f2661d2b34965736e", " Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c", "Successfully built registries-conf-ctl docopt", "Installing collected packages: docopt, toml, registries-conf-ctl", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-07-26T06:00:35.079 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:35 +0000 (0:00:04.977) 0:03:35.475 *********** 2021-07-26T06:00:35.254 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-07-26T06:00:35.256 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-07-26T06:00:35.280 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:35 +0000 (0:00:00.200) 0:03:35.676 *********** 2021-07-26T06:00:35.709 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-07-26T06:00:35.711 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1627279235.3338406-19442-106309793024925/source", "state": "file", "uid": 0} 2021-07-26T06:00:35.733 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:35 +0000 (0:00:00.452) 0:03:36.129 *********** 2021-07-26T06:00:35.949 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.037140", "end": "2021-07-26 06:00:35.926998", "rc": 0, "start": "2021-07-26 06:00:35.889858", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-07-26T06:00:35.976 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:35 +0000 (0:00:00.243) 0:03:36.372 *********** 2021-07-26T06:00:38.149 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-07-26T06:00:38.150 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-07-26 06:00:23 UTC", "ActiveEnterTimestampMonotonic": "226815807", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "docker.socket network-online.target firewalld.service sysinit.target system.slice systemd-journald.socket containerd.service basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-07-26 06:00:21 UTC", "AssertTimestampMonotonic": "224626224", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-07-26 06:00:21 UTC", "ConditionTimestampMonotonic": "224626224", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.sli 2021-07-26T06:00:38.152 INFO:teuthology.task.ansible.out:ce/docker.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8806", "ExecMainStartTimestamp": "Mon 2021-07-26 06:00:21 UTC", "ExecMainStartTimestampMonotonic": "224627046", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[Mon 2021-07-26 06:00:21 UTC] ; stop_time=[n/a] ; pid=8806 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[Mon 2021-07-26 06:00:21 UTC] ; stop_time=[n/a] ; pid=8806 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngress 2021-07-26T06:00:38.154 INFO:teuthology.task.ansible.out:Packets": "[no data]", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-07-26 06:00:21 UTC", "InactiveExitTimestampMonotonic": "224627311", "InvocationID": "4301cd43d9af4d779a996e332bba3302", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "8806", "MemoryAccounting": "yes", "MemoryCurrent": "53587968", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "contin 2021-07-26T06:00:38.155 INFO:teuthology.task.ansible.out:ue", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket sysinit.target system.slice", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-07-26 06:00:23 UTC", "StateChangeTimestampMonotonic": "226815807", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "15", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": " 2021-07-26T06:00:38.157 INFO:teuthology.task.ansible.out:notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "Wants": "containerd.service network-online.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-07-26T06:00:38.224 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-07-26T06:00:38.225 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-07-26T06:00:38.226 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-07-26T06:00:38.228 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-07-26T06:00:38.241 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Monday 26 July 2021 06:00:38 +0000 (0:00:02.264) 0:03:38.637 *********** 2021-07-26T06:00:39.204 INFO:teuthology.task.ansible.out:ok: [smithi094.front.sepia.ceph.com] 2021-07-26T06:00:39.226 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** 2021-07-26T06:00:39.228 INFO:teuthology.task.ansible.out:Monday 26 July 2021 06:00:39 +0000 (0:00:00.985) 0:03:39.622 *********** 2021-07-26T06:00:39.411 INFO:teuthology.task.ansible.out:changed: [smithi094.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-07-26T06:00:39.415 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi094.front.sepia.ceph.com : ok=133 changed=41 unreachable=0 failed=0 skipped=187 rescued=0 ignored=0 Monday 26 July 2021 06:00:39 +0000 (0:00:00.188) 0:03:39.811 *********** =============================================================================== 2021-07-26T06:00:39.417 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 27.04s testnode : Upgrade packages -------------------------------------------- 25.50s 2021-07-26T06:00:39.417 INFO:teuthology.task.ansible.out:users : Update authorized_keys using the keys repo --------------------- 19.74s container-host : Install container packages ---------------------------- 16.25s testnode : Install packages -------------------------------------------- 13.95s testnode : Ensure packages are not present. ---------------------------- 11.80s testnode : Zap all non-root disks --------------------------------------- 8.26s Remove revoked users ---------------------------------------------------- 6.91s container-host : Install registries-conf-ctl ---------------------------- 4.98s common : Update apt cache ----------------------------------------------- 4.01s testnode : Install packages via pip ------------------------------------- 2.40s 2021-07-26T06:00:39.418 INFO:teuthology.task.ansible.out:testnode : Create logical volume(s) ------------------------------------- 2.37s testnode : Install apt keys --------------------------------------------- 2.28s container-host : Restart docker service --------------------------------- 2.26s Gathering Facts --------------------------------------------------------- 2.23s testnode : Update apt cache. -------------------------------------------- 2.21s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.58s users : Update authorized_keys for each user with literal keys ---------- 1.46s common : Mask sleep units ----------------------------------------------- 1.24s testnode : Install python-pip on apt based systems. --------------------- 1.14s 2021-07-26T06:00:39.846 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com')] 2021-07-26T06:00:39.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T06:00:39.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-07-26T06:00:39.903 DEBUG:teuthology.orchestra.run.smithi094:> true 2021-07-26T06:00:40.024 INFO:teuthology.run_tasks:Running task clock... 2021-07-26T06:00:40.037 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-07-26T06:00:40.038 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-07-26T06:00:40.040 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-07-26T06:00:40.074 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-07-26T06:00:40.075 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Command line: ntpd -gq 2021-07-26T06:00:40.076 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: proto: precision = 0.265 usec (-22) 2021-07-26T06:00:40.076 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-07-26T06:00:40.076 INFO:teuthology.orchestra.run.smithi094.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-07-26T06:00:40.077 INFO:teuthology.orchestra.run.smithi094.stderr:restrict ::: KOD does nothing without LIMITED. 2021-07-26T06:00:40.078 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: restrict ::: KOD does nothing without LIMITED. 2021-07-26T06:00:40.078 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Listen and drop on 0 v6wildcard [::]:123 2021-07-26T06:00:40.078 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-07-26T06:00:40.079 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Listen normally on 2 lo 127.0.0.1:123 2021-07-26T06:00:40.079 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Listen normally on 3 enp3s0f1 172.21.15.94:123 2021-07-26T06:00:40.079 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Listen normally on 4 lo [::1]:123 2021-07-26T06:00:40.079 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d763%5]:123 2021-07-26T06:00:40.079 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:40 ntpd[10175]: Listening on routing socket on fd #22 for interface updates 2021-07-26T06:00:41.075 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:41 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:41.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:41 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:41.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:41 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:41.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:41 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:41.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:41 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:41.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:41 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:43.075 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:43 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:43 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:43 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:43 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:43.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:43 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:43.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:43 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:44.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:44.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:44.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:44 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:45.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:45.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:45.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:45.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:45.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:45.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:45.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:45.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:45.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:45.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:45.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:45.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:45 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:46.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:46.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:46.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:46.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:46.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:46.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:46.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:46.079 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:46.079 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:46 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:47.076 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:47.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:47.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:47.077 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:47.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-07-26T06:00:47.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: can't open /var/log/ntpstats/rawstats.20210726: Permission denied 2021-07-26T06:00:47.078 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-07-26T06:00:47.079 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: can't open /var/log/ntpstats/peerstats.20210726: Permission denied 2021-07-26T06:00:47.079 INFO:teuthology.orchestra.run.smithi094.stdout:26 Jul 06:00:47 ntpd[10175]: ntpd: time slew -0.000754 s 2021-07-26T06:00:47.080 INFO:teuthology.orchestra.run.smithi094.stdout:ntpd: time slew -0.000754s 2021-07-26T06:00:47.080 INFO:teuthology.orchestra.run.smithi094.stderr:26 Jul 06:00:47 ntpd[10175]: can't open /var/log/ntpstats/loopstats.20210726: Permission denied 2021-07-26T06:00:47.136 INFO:teuthology.orchestra.run.smithi094.stdout: remote refid st t when poll reach delay offset jitter 2021-07-26T06:00:47.137 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================== 2021-07-26T06:00:47.137 INFO:teuthology.orchestra.run.smithi094.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-07-26T06:00:47.137 INFO:teuthology.orchestra.run.smithi094.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-07-26T06:00:47.138 INFO:teuthology.orchestra.run.smithi094.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-07-26T06:00:47.138 INFO:teuthology.orchestra.run.smithi094.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-07-26T06:00:47.139 INFO:teuthology.run_tasks:Running task install... 2021-07-26T06:00:47.154 DEBUG:teuthology.task.install:project ceph 2021-07-26T06:00:47.155 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '197e564fdaee12596de1dc8f3253d1704ae66d5d'}} 2021-07-26T06:00:47.155 DEBUG:teuthology.task.install:config {'sha1': '197e564fdaee12596de1dc8f3253d1704ae66d5d'} 2021-07-26T06:00:47.156 INFO:teuthology.task.install:Using flavor: basic 2021-07-26T06:00:47.163 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-07-26T06:00:47.163 INFO:teuthology.task.install:extra packages: [] 2021-07-26T06:00:47.164 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-07-26T06:00:47.226 INFO:teuthology.orchestra.run.smithi094.stdout:Ubuntu 2021-07-26T06:00:47.231 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-07-26T06:00:47.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-key list | grep Ceph 2021-07-26T06:00:47.328 INFO:teuthology.orchestra.run.smithi094.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-07-26T06:00:47.388 INFO:teuthology.orchestra.run.smithi094.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-07-26T06:00:47.388 INFO:teuthology.orchestra.run.smithi094.stdout:uid [ unknown] Ceph.com (release key) 2021-07-26T06:00:47.390 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-07-26T06:00:47.439 INFO:teuthology.orchestra.run.smithi094.stdout:Ubuntu 2021-07-26T06:00:47.444 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-07-26T06:00:47.444 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-07-26T06:00:47.445 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:00:47.708 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default/ 2021-07-26T06:00:47.708 INFO:teuthology.task.install.deb:Package version is 17.0.0-6358-g197e564f-1focal 2021-07-26T06:00:47.947 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-07-26T06:00:47.948 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-07-26T06:00:47.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo apt-get update 2021-07-26T06:00:48.149 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-07-26T06:00:48.308 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-07-26T06:00:48.333 INFO:teuthology.orchestra.run.smithi094.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal InRelease 2021-07-26T06:00:48.386 INFO:teuthology.orchestra.run.smithi094.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal Release [7,662 B] 2021-07-26T06:00:48.415 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-07-26T06:00:48.432 INFO:teuthology.orchestra.run.smithi094.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal Release.gpg 2021-07-26T06:00:48.479 INFO:teuthology.orchestra.run.smithi094.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main i386 Packages [3,110 B] 2021-07-26T06:00:48.537 INFO:teuthology.orchestra.run.smithi094.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-07-26T06:00:48.549 INFO:teuthology.orchestra.run.smithi094.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 Packages [17.1 kB] 2021-07-26T06:00:48.822 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 27.9 kB in 1s (43.1 kB/s) 2021-07-26T06:00:49.592 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T06:00:49.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-6358-g197e564f-1focal cephadm=17.0.0-6358-g197e564f-1focal ceph-mds=17.0.0-6358-g197e564f-1focal ceph-mgr=17.0.0-6358-g197e564f-1focal ceph-common=17.0.0-6358-g197e564f-1focal ceph-fuse=17.0.0-6358-g197e564f-1focal ceph-test=17.0.0-6358-g197e564f-1focal radosgw=17.0.0-6358-g197e564f-1focal python3-rados=17.0.0-6358-g197e564f-1focal python3-rgw=17.0.0-6358-g197e564f-1focal python3-cephfs=17.0.0-6358-g197e564f-1focal python3-rbd=17.0.0-6358-g197e564f-1focal libcephfs2=17.0.0-6358-g197e564f-1focal libcephfs-dev=17.0.0-6358-g197e564f-1focal librados2=17.0.0-6358-g197e564f-1focal librbd1=17.0.0-6358-g197e564f-1focal rbd-fuse=17.0.0-6358-g197e564f-1focal 2021-07-26T06:00:49.692 INFO:teuthology.orchestra.run.smithi094.stdout:Reading package lists... 2021-07-26T06:00:49.835 INFO:teuthology.orchestra.run.smithi094.stdout:Building dependency tree... 2021-07-26T06:00:49.836 INFO:teuthology.orchestra.run.smithi094.stdout:Reading state information... 2021-07-26T06:00:49.957 INFO:teuthology.orchestra.run.smithi094.stdout:The following packages were automatically installed and are no longer required: 2021-07-26T06:00:49.964 INFO:teuthology.orchestra.run.smithi094.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-07-26T06:00:49.964 INFO:teuthology.orchestra.run.smithi094.stdout:Use 'sudo apt autoremove' to remove them. 2021-07-26T06:00:49.965 INFO:teuthology.orchestra.run.smithi094.stdout:The following additional packages will be installed: 2021-07-26T06:00:49.965 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-07-26T06:00:49.965 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2021-07-26T06:00:49.965 INFO:teuthology.orchestra.run.smithi094.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d liblttng-ust0 2021-07-26T06:00:49.966 INFO:teuthology.orchestra.run.smithi094.stdout: liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2021-07-26T06:00:49.966 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2021-07-26T06:00:49.966 INFO:teuthology.orchestra.run.smithi094.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2021-07-26T06:00:49.966 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2021-07-26T06:00:49.967 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2021-07-26T06:00:49.967 INFO:teuthology.orchestra.run.smithi094.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2021-07-26T06:00:49.967 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2021-07-26T06:00:49.967 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-07-26T06:00:49.967 INFO:teuthology.orchestra.run.smithi094.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-07-26T06:00:49.968 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-07-26T06:00:49.968 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2021-07-26T06:00:49.968 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-07-26T06:00:49.968 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-07-26T06:00:49.968 INFO:teuthology.orchestra.run.smithi094.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-07-26T06:00:49.969 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-07-26T06:00:49.969 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-07-26T06:00:49.969 INFO:teuthology.orchestra.run.smithi094.stdout:Suggested packages: 2021-07-26T06:00:49.969 INFO:teuthology.orchestra.run.smithi094.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-07-26T06:00:49.970 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-07-26T06:00:49.970 INFO:teuthology.orchestra.run.smithi094.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-07-26T06:00:49.970 INFO:teuthology.orchestra.run.smithi094.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-07-26T06:00:49.970 INFO:teuthology.orchestra.run.smithi094.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-07-26T06:00:49.970 INFO:teuthology.orchestra.run.smithi094.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-07-26T06:00:49.971 INFO:teuthology.orchestra.run.smithi094.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-07-26T06:00:49.971 INFO:teuthology.orchestra.run.smithi094.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-07-26T06:00:49.971 INFO:teuthology.orchestra.run.smithi094.stdout:Recommended packages: 2021-07-26T06:00:49.971 INFO:teuthology.orchestra.run.smithi094.stdout: btrfs-tools 2021-07-26T06:00:50.046 INFO:teuthology.orchestra.run.smithi094.stdout:The following NEW packages will be installed: 2021-07-26T06:00:50.046 INFO:teuthology.orchestra.run.smithi094.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-07-26T06:00:50.047 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-07-26T06:00:50.047 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-07-26T06:00:50.047 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-07-26T06:00:50.047 INFO:teuthology.orchestra.run.smithi094.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-07-26T06:00:50.048 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2021-07-26T06:00:50.048 INFO:teuthology.orchestra.run.smithi094.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-07-26T06:00:50.048 INFO:teuthology.orchestra.run.smithi094.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-07-26T06:00:50.048 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-07-26T06:00:50.048 INFO:teuthology.orchestra.run.smithi094.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-07-26T06:00:50.049 INFO:teuthology.orchestra.run.smithi094.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2021-07-26T06:00:50.049 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2021-07-26T06:00:50.049 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2021-07-26T06:00:50.049 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2021-07-26T06:00:50.050 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2021-07-26T06:00:50.050 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2021-07-26T06:00:50.050 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-07-26T06:00:50.051 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-07-26T06:00:50.051 INFO:teuthology.orchestra.run.smithi094.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-07-26T06:00:50.051 INFO:teuthology.orchestra.run.smithi094.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-07-26T06:00:50.051 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-07-26T06:00:50.052 INFO:teuthology.orchestra.run.smithi094.stdout:The following packages will be upgraded: 2021-07-26T06:00:50.052 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 librbd1 2021-07-26T06:00:50.209 INFO:teuthology.orchestra.run.smithi094.stdout:2 upgraded, 98 newly installed, 0 to remove and 143 not upgraded. 2021-07-26T06:00:50.210 INFO:teuthology.orchestra.run.smithi094.stdout:Need to get 181 MB of archives. 2021-07-26T06:00:50.210 INFO:teuthology.orchestra.run.smithi094.stdout:After this operation, 845 MB of additional disk space will be used. 2021-07-26T06:00:50.210 INFO:teuthology.orchestra.run.smithi094.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-6358-g197e564f-1focal [3,470 kB] 2021-07-26T06:00:50.239 INFO:teuthology.orchestra.run.smithi094.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-07-26T06:00:50.423 INFO:teuthology.orchestra.run.smithi094.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-6358-g197e564f-1focal [3,494 kB] 2021-07-26T06:00:50.529 INFO:teuthology.orchestra.run.smithi094.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-6358-g197e564f-1focal [743 kB] 2021-07-26T06:00:50.553 INFO:teuthology.orchestra.run.smithi094.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-6358-g197e564f-1focal [408 kB] 2021-07-26T06:00:50.566 INFO:teuthology.orchestra.run.smithi094.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-6358-g197e564f-1focal [22.4 kB] 2021-07-26T06:00:50.567 INFO:teuthology.orchestra.run.smithi094.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-6358-g197e564f-1focal [200 kB] 2021-07-26T06:00:50.575 INFO:teuthology.orchestra.run.smithi094.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-6358-g197e564f-1focal [31.1 kB] 2021-07-26T06:00:50.576 INFO:teuthology.orchestra.run.smithi094.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-6358-g197e564f-1focal [384 kB] 2021-07-26T06:00:50.587 INFO:teuthology.orchestra.run.smithi094.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-6358-g197e564f-1focal [4,150 kB] 2021-07-26T06:00:50.610 INFO:teuthology.orchestra.run.smithi094.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-07-26T06:00:50.611 INFO:teuthology.orchestra.run.smithi094.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-07-26T06:00:50.616 INFO:teuthology.orchestra.run.smithi094.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-07-26T06:00:50.694 INFO:teuthology.orchestra.run.smithi094.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-07-26T06:00:50.725 INFO:teuthology.orchestra.run.smithi094.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-6358-g197e564f-1focal [116 kB] 2021-07-26T06:00:50.730 INFO:teuthology.orchestra.run.smithi094.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-6358-g197e564f-1focal [477 kB] 2021-07-26T06:00:50.744 INFO:teuthology.orchestra.run.smithi094.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-6358-g197e564f-1focal [23.3 MB] 2021-07-26T06:00:50.800 INFO:teuthology.orchestra.run.smithi094.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-07-26T06:00:50.818 INFO:teuthology.orchestra.run.smithi094.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-07-26T06:00:50.825 INFO:teuthology.orchestra.run.smithi094.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-07-26T06:00:50.856 INFO:teuthology.orchestra.run.smithi094.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-07-26T06:00:50.861 INFO:teuthology.orchestra.run.smithi094.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-07-26T06:00:50.863 INFO:teuthology.orchestra.run.smithi094.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-07-26T06:00:50.880 INFO:teuthology.orchestra.run.smithi094.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-07-26T06:00:50.881 INFO:teuthology.orchestra.run.smithi094.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-07-26T06:00:50.883 INFO:teuthology.orchestra.run.smithi094.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-07-26T06:00:50.940 INFO:teuthology.orchestra.run.smithi094.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-07-26T06:00:50.944 INFO:teuthology.orchestra.run.smithi094.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-07-26T06:00:50.944 INFO:teuthology.orchestra.run.smithi094.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-07-26T06:00:50.945 INFO:teuthology.orchestra.run.smithi094.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-07-26T06:00:51.022 INFO:teuthology.orchestra.run.smithi094.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-07-26T06:00:51.023 INFO:teuthology.orchestra.run.smithi094.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-07-26T06:00:51.027 INFO:teuthology.orchestra.run.smithi094.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-07-26T06:00:51.030 INFO:teuthology.orchestra.run.smithi094.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-07-26T06:00:51.031 INFO:teuthology.orchestra.run.smithi094.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-07-26T06:00:51.052 INFO:teuthology.orchestra.run.smithi094.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-07-26T06:00:51.104 INFO:teuthology.orchestra.run.smithi094.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-07-26T06:00:51.105 INFO:teuthology.orchestra.run.smithi094.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-07-26T06:00:51.106 INFO:teuthology.orchestra.run.smithi094.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-07-26T06:00:51.110 INFO:teuthology.orchestra.run.smithi094.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-07-26T06:00:51.188 INFO:teuthology.orchestra.run.smithi094.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-07-26T06:00:51.192 INFO:teuthology.orchestra.run.smithi094.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-07-26T06:00:51.193 INFO:teuthology.orchestra.run.smithi094.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2021-07-26T06:00:51.197 INFO:teuthology.orchestra.run.smithi094.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-07-26T06:00:51.200 INFO:teuthology.orchestra.run.smithi094.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-07-26T06:00:51.203 INFO:teuthology.orchestra.run.smithi094.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-07-26T06:00:51.270 INFO:teuthology.orchestra.run.smithi094.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-07-26T06:00:51.272 INFO:teuthology.orchestra.run.smithi094.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-07-26T06:00:51.607 INFO:teuthology.orchestra.run.smithi094.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-6358-g197e564f-1focal [6,368 kB] 2021-07-26T06:00:51.672 INFO:teuthology.orchestra.run.smithi094.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-07-26T06:00:51.740 INFO:teuthology.orchestra.run.smithi094.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-07-26T06:00:51.740 INFO:teuthology.orchestra.run.smithi094.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-6358-g197e564f-1focal [196 kB] 2021-07-26T06:00:51.746 INFO:teuthology.orchestra.run.smithi094.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-6358-g197e564f-1focal [151 kB] 2021-07-26T06:00:51.748 INFO:teuthology.orchestra.run.smithi094.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-07-26T06:00:51.841 INFO:teuthology.orchestra.run.smithi094.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-6358-g197e564f-1focal [1,548 kB] 2021-07-26T06:00:51.841 INFO:teuthology.orchestra.run.smithi094.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-07-26T06:00:51.841 INFO:teuthology.orchestra.run.smithi094.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-07-26T06:00:51.842 INFO:teuthology.orchestra.run.smithi094.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-07-26T06:00:51.843 INFO:teuthology.orchestra.run.smithi094.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-07-26T06:00:51.844 INFO:teuthology.orchestra.run.smithi094.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-07-26T06:00:51.845 INFO:teuthology.orchestra.run.smithi094.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-07-26T06:00:51.845 INFO:teuthology.orchestra.run.smithi094.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-07-26T06:00:51.858 INFO:teuthology.orchestra.run.smithi094.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-6358-g197e564f-1focal [7,804 kB] 2021-07-26T06:00:51.874 INFO:teuthology.orchestra.run.smithi094.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-07-26T06:00:51.874 INFO:teuthology.orchestra.run.smithi094.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-07-26T06:00:51.965 INFO:teuthology.orchestra.run.smithi094.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-07-26T06:00:51.970 INFO:teuthology.orchestra.run.smithi094.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-07-26T06:00:51.973 INFO:teuthology.orchestra.run.smithi094.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-07-26T06:00:51.978 INFO:teuthology.orchestra.run.smithi094.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-07-26T06:00:51.979 INFO:teuthology.orchestra.run.smithi094.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-07-26T06:00:51.981 INFO:teuthology.orchestra.run.smithi094.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-07-26T06:00:51.982 INFO:teuthology.orchestra.run.smithi094.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-07-26T06:00:51.984 INFO:teuthology.orchestra.run.smithi094.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-07-26T06:00:52.047 INFO:teuthology.orchestra.run.smithi094.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-07-26T06:00:52.048 INFO:teuthology.orchestra.run.smithi094.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-07-26T06:00:52.074 INFO:teuthology.orchestra.run.smithi094.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-6358-g197e564f-1focal [28.4 MB] 2021-07-26T06:00:52.129 INFO:teuthology.orchestra.run.smithi094.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-07-26T06:00:52.129 INFO:teuthology.orchestra.run.smithi094.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-07-26T06:00:52.131 INFO:teuthology.orchestra.run.smithi094.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.3 [1,382 kB] 2021-07-26T06:00:52.151 INFO:teuthology.orchestra.run.smithi094.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-07-26T06:00:52.151 INFO:teuthology.orchestra.run.smithi094.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-07-26T06:00:52.152 INFO:teuthology.orchestra.run.smithi094.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-07-26T06:00:52.152 INFO:teuthology.orchestra.run.smithi094.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-07-26T06:00:52.153 INFO:teuthology.orchestra.run.smithi094.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-07-26T06:00:52.211 INFO:teuthology.orchestra.run.smithi094.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-07-26T06:00:52.211 INFO:teuthology.orchestra.run.smithi094.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-07-26T06:00:52.293 INFO:teuthology.orchestra.run.smithi094.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-07-26T06:00:52.296 INFO:teuthology.orchestra.run.smithi094.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-07-26T06:00:53.131 INFO:teuthology.orchestra.run.smithi094.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-6358-g197e564f-1focal [3,812 B] 2021-07-26T06:00:53.132 INFO:teuthology.orchestra.run.smithi094.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-6358-g197e564f-1focal [850 kB] 2021-07-26T06:00:53.147 INFO:teuthology.orchestra.run.smithi094.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-6358-g197e564f-1focal [2,350 kB] 2021-07-26T06:00:53.164 INFO:teuthology.orchestra.run.smithi094.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-6358-g197e564f-1focal [66.3 kB] 2021-07-26T06:00:53.164 INFO:teuthology.orchestra.run.smithi094.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-6358-g197e564f-1focal [85.4 kB] 2021-07-26T06:00:53.165 INFO:teuthology.orchestra.run.smithi094.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-6358-g197e564f-1focal [1,591 kB] 2021-07-26T06:00:53.180 INFO:teuthology.orchestra.run.smithi094.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-6358-g197e564f-1focal [7,871 kB] 2021-07-26T06:00:53.435 INFO:teuthology.orchestra.run.smithi094.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-6358-g197e564f-1focal [3,928 B] 2021-07-26T06:00:53.435 INFO:teuthology.orchestra.run.smithi094.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-6358-g197e564f-1focal [52.2 MB] 2021-07-26T06:00:55.438 INFO:teuthology.orchestra.run.smithi094.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-6358-g197e564f-1focal [20.8 kB] 2021-07-26T06:00:55.439 INFO:teuthology.orchestra.run.smithi094.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-6358-g197e564f-1focal [12.4 MB] 2021-07-26T06:00:55.729 INFO:teuthology.orchestra.run.smithi094.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-07-26-1205/197e564fdaee12596de1dc8f3253d1704ae66d5d/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-6358-g197e564f-1focal [78.8 kB] 2021-07-26T06:00:56.305 INFO:teuthology.orchestra.run.smithi094.stdout:Fetched 181 MB in 6s (31.9 MB/s) 2021-07-26T06:00:56.474 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-07-26T06:00:56.528 INFO:teuthology.orchestra.run.smithi094.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138107 files and directories currently installed.) 2021-07-26T06:00:56.532 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-07-26T06:00:56.615 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-07-26T06:00:56.984 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../01-librbd1_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:00:57.143 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librbd1 (17.0.0-6358-g197e564f-1focal) over (15.2.13-0ubuntu0.20.04.1) ... 2021-07-26T06:00:57.833 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../02-librados2_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:00:57.953 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librados2 (17.0.0-6358-g197e564f-1focal) over (15.2.13-0ubuntu0.20.04.1) ... 2021-07-26T06:00:58.643 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libcephfs2. 2021-07-26T06:00:58.658 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:00:58.701 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libcephfs2 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:00:59.078 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rados. 2021-07-26T06:00:59.097 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../04-python3-rados_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:00:59.137 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rados (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:00:59.456 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-07-26T06:00:59.474 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-6358-g197e564f-1focal_all.deb ... 2021-07-26T06:00:59.524 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-ceph-argparse (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:00:59.734 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cephfs. 2021-07-26T06:00:59.753 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:00:59.817 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cephfs (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:00.169 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-ceph-common. 2021-07-26T06:01:00.186 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-6358-g197e564f-1focal_all.deb ... 2021-07-26T06:01:00.227 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-ceph-common (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:00.564 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-prettytable. 2021-07-26T06:01:00.579 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2021-07-26T06:01:00.622 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-07-26T06:01:00.891 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rbd. 2021-07-26T06:01:00.906 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:00.941 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rbd (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:01.252 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-07-26T06:01:01.267 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-07-26T06:01:01.335 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-07-26T06:01:01.721 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-07-26T06:01:01.737 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2021-07-26T06:01:01.780 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-07-26T06:01:02.132 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-07-26T06:01:02.150 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-07-26T06:01:02.195 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-07-26T06:01:02.536 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package librgw2. 2021-07-26T06:01:02.553 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../13-librgw2_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:02.585 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking librgw2 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:03.254 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rgw. 2021-07-26T06:01:03.273 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:03.312 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rgw (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:03.665 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-07-26T06:01:03.682 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../15-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-07-26T06:01:03.750 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-07-26T06:01:04.094 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package liboath0. 2021-07-26T06:01:04.109 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../16-liboath0_2.6.1-1.3_amd64.deb ... 2021-07-26T06:01:04.143 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-07-26T06:01:04.404 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libradosstriper1. 2021-07-26T06:01:04.419 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../17-libradosstriper1_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:04.455 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libradosstriper1 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:04.973 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-common. 2021-07-26T06:01:04.990 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../18-ceph-common_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:05.033 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-common (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:07.409 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-base. 2021-07-26T06:01:07.421 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../19-ceph-base_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:07.474 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-base (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:08.516 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cryptography. 2021-07-26T06:01:08.531 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../20-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-07-26T06:01:08.607 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-07-26T06:01:09.403 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-openssl. 2021-07-26T06:01:09.418 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../21-python3-openssl_19.0.0-1build1_all.deb ... 2021-07-26T06:01:09.461 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-07-26T06:01:09.755 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-07-26T06:01:09.774 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../22-ceph-mgr-modules-core_17.0.0-6358-g197e564f-1focal_all.deb ... 2021-07-26T06:01:09.813 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-modules-core (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:10.235 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-bcrypt. 2021-07-26T06:01:10.253 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../23-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-07-26T06:01:10.293 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-07-26T06:01:10.629 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cherrypy3. 2021-07-26T06:01:10.645 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../24-python3-cherrypy3_8.9.1-6_all.deb ... 2021-07-26T06:01:10.688 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-07-26T06:01:11.098 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-jwt. 2021-07-26T06:01:11.116 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../25-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-07-26T06:01:11.156 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-07-26T06:01:11.486 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-logutils. 2021-07-26T06:01:11.501 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../26-python3-logutils_0.3.3-7_all.deb ... 2021-07-26T06:01:11.535 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-07-26T06:01:11.804 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-markupsafe. 2021-07-26T06:01:11.819 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../27-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-07-26T06:01:11.854 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-07-26T06:01:12.173 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-mako. 2021-07-26T06:01:12.192 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../28-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-07-26T06:01:12.253 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-07-26T06:01:12.610 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-simplegeneric. 2021-07-26T06:01:12.626 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../29-python3-simplegeneric_0.8.1-3_all.deb ... 2021-07-26T06:01:12.675 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-07-26T06:01:13.004 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-singledispatch. 2021-07-26T06:01:13.021 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../30-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-07-26T06:01:13.062 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-07-26T06:01:13.356 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-webob. 2021-07-26T06:01:13.375 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../31-python3-webob_1%3a1.8.5-2_all.deb ... 2021-07-26T06:01:13.414 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-07-26T06:01:13.750 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-soupsieve. 2021-07-26T06:01:13.767 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../32-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-07-26T06:01:13.834 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-07-26T06:01:14.078 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-bs4. 2021-07-26T06:01:14.094 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../33-python3-bs4_4.8.2-1_all.deb ... 2021-07-26T06:01:14.161 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-07-26T06:01:14.473 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-waitress. 2021-07-26T06:01:14.492 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../34-python3-waitress_1.4.1-1_all.deb ... 2021-07-26T06:01:14.534 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-07-26T06:01:14.902 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-tempita. 2021-07-26T06:01:14.920 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../35-python3-tempita_0.5.2-6_all.deb ... 2021-07-26T06:01:14.959 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-07-26T06:01:15.287 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-paste. 2021-07-26T06:01:15.302 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../36-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-07-26T06:01:15.345 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-07-26T06:01:15.690 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-07-26T06:01:15.705 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../37-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-07-26T06:01:15.781 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-07-26T06:01:16.109 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pastedeploy. 2021-07-26T06:01:16.126 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../38-python3-pastedeploy_2.1.0-1_all.deb ... 2021-07-26T06:01:16.176 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-07-26T06:01:16.503 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-webtest. 2021-07-26T06:01:16.519 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../39-python3-webtest_2.0.32-2_all.deb ... 2021-07-26T06:01:16.561 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-07-26T06:01:16.914 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pecan. 2021-07-26T06:01:16.933 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../40-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-07-26T06:01:16.972 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-07-26T06:01:17.394 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-werkzeug. 2021-07-26T06:01:17.412 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../41-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-07-26T06:01:17.452 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-07-26T06:01:17.797 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2021-07-26T06:01:17.816 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../42-libsqlite3-mod-ceph_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:17.855 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:18.270 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr. 2021-07-26T06:01:18.288 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../43-ceph-mgr_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:18.332 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:18.844 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mon. 2021-07-26T06:01:18.859 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../44-ceph-mon_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:18.927 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mon (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:20.048 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-osd. 2021-07-26T06:01:20.063 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../45-ceph-osd_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:20.134 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-osd (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:23.044 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph. 2021-07-26T06:01:23.063 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../46-ceph_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:23.102 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:23.455 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-fuse. 2021-07-26T06:01:23.470 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../47-ceph-fuse_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:23.513 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-fuse (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:23.941 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mds. 2021-07-26T06:01:23.957 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../48-ceph-mds_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:23.991 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mds (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:25.197 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package cephadm. 2021-07-26T06:01:25.216 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../49-cephadm_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:25.277 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking cephadm (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:25.616 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-jinja2. 2021-07-26T06:01:25.632 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../50-python3-jinja2_2.10.1-2_all.deb ... 2021-07-26T06:01:25.674 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-07-26T06:01:26.039 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-07-26T06:01:26.059 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../51-ceph-mgr-cephadm_17.0.0-6358-g197e564f-1focal_all.deb ... 2021-07-26T06:01:26.097 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-cephadm (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:26.457 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-repoze.lru. 2021-07-26T06:01:26.473 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../52-python3-repoze.lru_0.7-2_all.deb ... 2021-07-26T06:01:26.527 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-07-26T06:01:26.785 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libjs-underscore. 2021-07-26T06:01:26.802 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../53-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2021-07-26T06:01:26.871 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-07-26T06:01:27.188 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-07-26T06:01:27.203 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../54-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-07-26T06:01:27.246 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-07-26T06:01:27.574 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-routes. 2021-07-26T06:01:27.592 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../55-python3-routes_2.4.1-2_all.deb ... 2021-07-26T06:01:27.633 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-07-26T06:01:27.986 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-07-26T06:01:28.002 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../56-ceph-mgr-dashboard_17.0.0-6358-g197e564f-1focal_all.deb ... 2021-07-26T06:01:28.053 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-dashboard (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:28.549 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-decorator. 2021-07-26T06:01:28.565 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../57-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-07-26T06:01:28.607 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-07-26T06:01:28.935 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-07-26T06:01:28.951 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../58-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-07-26T06:01:28.993 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-07-26T06:01:29.297 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-scipy. 2021-07-26T06:01:29.314 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../59-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-07-26T06:01:29.372 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-07-26T06:01:30.782 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-07-26T06:01:30.799 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../60-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-07-26T06:01:30.865 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-07-26T06:01:31.421 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-joblib. 2021-07-26T06:01:31.436 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../61-python3-joblib_0.14.0-3_all.deb ... 2021-07-26T06:01:31.520 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-07-26T06:01:31.899 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-sklearn. 2021-07-26T06:01:31.917 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../62-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-07-26T06:01:31.957 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-07-26T06:01:32.767 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-07-26T06:01:32.786 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../63-ceph-mgr-diskprediction-local_17.0.0-6358-g197e564f-1focal_all.deb ... 2021-07-26T06:01:32.825 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:34.004 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-cachetools. 2021-07-26T06:01:34.022 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../64-python3-cachetools_4.0.0-1_all.deb ... 2021-07-26T06:01:34.062 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-07-26T06:01:34.341 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pyasn1. 2021-07-26T06:01:34.358 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../65-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-07-26T06:01:34.391 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-07-26T06:01:34.704 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-07-26T06:01:34.724 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../66-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-07-26T06:01:34.762 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-07-26T06:01:35.100 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-rsa. 2021-07-26T06:01:35.119 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../67-python3-rsa_4.0-3_all.deb ... 2021-07-26T06:01:35.157 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-rsa (4.0-3) ... 2021-07-26T06:01:35.503 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-google-auth. 2021-07-26T06:01:35.519 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../68-python3-google-auth_1.5.1-2_all.deb ... 2021-07-26T06:01:35.578 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-07-26T06:01:35.898 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-blinker. 2021-07-26T06:01:35.917 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../69-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-07-26T06:01:35.950 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-07-26T06:01:36.235 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-oauthlib. 2021-07-26T06:01:36.252 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../70-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-07-26T06:01:36.293 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-07-26T06:01:36.672 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-07-26T06:01:36.690 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../71-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-07-26T06:01:36.747 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-07-26T06:01:37.109 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-websocket. 2021-07-26T06:01:37.126 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../72-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-07-26T06:01:37.167 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-07-26T06:01:37.487 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-kubernetes. 2021-07-26T06:01:37.505 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../73-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-07-26T06:01:37.545 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-07-26T06:01:38.118 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-07-26T06:01:38.137 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../74-ceph-mgr-k8sevents_17.0.0-6358-g197e564f-1focal_all.deb ... 2021-07-26T06:01:38.177 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:38.524 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libonig5:amd64. 2021-07-26T06:01:38.543 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../75-libonig5_6.9.4-1_amd64.deb ... 2021-07-26T06:01:38.598 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-07-26T06:01:38.900 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libjq1:amd64. 2021-07-26T06:01:38.920 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../76-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-07-26T06:01:38.983 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-07-26T06:01:39.286 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package jq. 2021-07-26T06:01:39.302 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../77-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-07-26T06:01:39.344 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-07-26T06:01:39.640 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package socat. 2021-07-26T06:01:39.659 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../78-socat_1.7.3.3-2_amd64.deb ... 2021-07-26T06:01:39.698 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking socat (1.7.3.3-2) ... 2021-07-26T06:01:40.060 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-07-26T06:01:40.078 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../79-libxslt1.1_1.1.34-4_amd64.deb ... 2021-07-26T06:01:40.110 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-07-26T06:01:41.014 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package xmlstarlet. 2021-07-26T06:01:41.030 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../80-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-07-26T06:01:41.072 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-07-26T06:01:41.537 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package ceph-test. 2021-07-26T06:01:41.553 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../81-ceph-test_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:41.595 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking ceph-test (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:46.017 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package libcephfs-dev. 2021-07-26T06:01:46.032 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../82-libcephfs-dev_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:46.225 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking libcephfs-dev (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:46.645 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package nvme-cli. 2021-07-26T06:01:46.662 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../83-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-07-26T06:01:46.721 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-07-26T06:01:47.085 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-atomicwrites. 2021-07-26T06:01:47.104 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../84-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-07-26T06:01:47.159 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-07-26T06:01:47.454 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-attr. 2021-07-26T06:01:47.470 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../85-python3-attr_19.3.0-2_all.deb ... 2021-07-26T06:01:47.512 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-07-26T06:01:47.875 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-webencodings. 2021-07-26T06:01:47.893 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../86-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-07-26T06:01:47.932 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-07-26T06:01:48.286 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-html5lib. 2021-07-26T06:01:48.302 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../87-python3-html5lib_1.0.1-2_all.deb ... 2021-07-26T06:01:48.386 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-07-26T06:01:48.773 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-07-26T06:01:48.792 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../88-python3-lxml_4.5.0-1ubuntu0.3_amd64.deb ... 2021-07-26T06:01:48.831 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-07-26T06:01:49.360 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-packaging. 2021-07-26T06:01:49.377 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../89-python3-packaging_20.3-1_all.deb ... 2021-07-26T06:01:49.418 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-packaging (20.3-1) ... 2021-07-26T06:01:49.747 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pastescript. 2021-07-26T06:01:49.762 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../90-python3-pastescript_2.0.2-4_all.deb ... 2021-07-26T06:01:49.813 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-07-26T06:01:50.193 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pluggy. 2021-07-26T06:01:50.209 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../91-python3-pluggy_0.13.0-2_all.deb ... 2021-07-26T06:01:50.276 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-07-26T06:01:50.621 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-psutil. 2021-07-26T06:01:50.637 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../92-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-07-26T06:01:50.679 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-07-26T06:01:51.033 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-py. 2021-07-26T06:01:51.052 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../93-python3-py_1.8.1-1_all.deb ... 2021-07-26T06:01:51.091 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-py (1.8.1-1) ... 2021-07-26T06:01:51.428 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pyinotify. 2021-07-26T06:01:51.447 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../94-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-07-26T06:01:51.511 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-07-26T06:01:51.873 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-wcwidth. 2021-07-26T06:01:51.890 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../95-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-07-26T06:01:51.931 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-07-26T06:01:52.268 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-pytest. 2021-07-26T06:01:52.285 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../96-python3-pytest_4.6.9-1_all.deb ... 2021-07-26T06:01:52.330 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-07-26T06:01:52.677 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package python3-simplejson. 2021-07-26T06:01:52.696 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../97-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-07-26T06:01:52.760 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-07-26T06:01:53.256 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package radosgw. 2021-07-26T06:01:53.275 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../98-radosgw_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:53.342 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking radosgw (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:54.757 INFO:teuthology.orchestra.run.smithi094.stdout:Selecting previously unselected package rbd-fuse. 2021-07-26T06:01:54.776 INFO:teuthology.orchestra.run.smithi094.stdout:Preparing to unpack .../99-rbd-fuse_17.0.0-6358-g197e564f-1focal_amd64.deb ... 2021-07-26T06:01:54.815 INFO:teuthology.orchestra.run.smithi094.stdout:Unpacking rbd-fuse (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:55.152 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-attr (19.3.0-2) ... 2021-07-26T06:01:55.469 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-07-26T06:01:55.635 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-07-26T06:01:55.762 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-07-26T06:01:55.989 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up cephadm (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:56.618 INFO:teuthology.orchestra.run.smithi094.stdout:Adding system user cephadm....done 2021-07-26T06:01:56.746 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-07-26T06:01:57.131 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-07-26T06:01:57.403 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-07-26T06:01:57.524 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-py (1.8.1-1) ... 2021-07-26T06:01:57.825 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-07-26T06:01:58.228 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-07-26T06:01:58.546 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-07-26T06:01:58.840 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-07-26T06:01:59.141 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-07-26T06:01:59.450 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-ceph-argparse (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:01:59.819 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-07-26T06:01:59.945 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-07-26T06:02:00.238 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-07-26T06:02:00.507 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-07-26T06:02:00.893 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-07-26T06:02:01.169 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-07-26T06:02:01.521 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-07-26T06:02:01.856 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-07-26T06:02:02.242 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-07-26T06:02:02.556 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-packaging (20.3-1) ... 2021-07-26T06:02:02.877 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-07-26T06:02:03.003 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-07-26T06:02:03.296 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-07-26T06:02:03.742 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-07-26T06:02:04.109 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-07-26T06:02:04.478 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up socat (1.7.3.3-2) ... 2021-07-26T06:02:04.604 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-ceph-common (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:04.939 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-07-26T06:02:05.349 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-07-26T06:02:05.478 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-07-26T06:02:05.756 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-07-26T06:02:05.883 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-07-26T06:02:06.034 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-07-26T06:02:06.345 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-07-26T06:02:06.629 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-07-26T06:02:06.939 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-07-26T06:02:07.107 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-07-26T06:02:07.408 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-07-26T06:02:07.726 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-07-26T06:02:08.053 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-07-26T06:02:08.412 INFO:teuthology.orchestra.run.smithi094.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-07-26T06:02:08.681 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-07-26T06:02:08.807 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-07-26T06:02:09.125 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-07-26T06:02:09.469 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-07-26T06:02:09.620 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-07-26T06:02:09.747 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-07-26T06:02:10.090 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-07-26T06:02:10.417 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librados2 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:10.576 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-07-26T06:02:12.041 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librgw2 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:12.167 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libsqlite3-mod-ceph (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:12.320 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-07-26T06:02:12.463 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-07-26T06:02:12.898 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-07-26T06:02:13.200 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-07-26T06:02:13.501 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-07-26T06:02:13.853 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libcephfs2 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:13.987 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-07-26T06:02:14.371 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libradosstriper1 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:14.500 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-07-26T06:02:14.823 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-07-26T06:02:15.863 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up librbd1 (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:16.023 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-07-26T06:02:16.149 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-07-26T06:02:16.302 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-07-26T06:02:16.627 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-07-26T06:02:16.753 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rsa (4.0-3) ... 2021-07-26T06:02:17.072 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-07-26T06:02:17.406 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-modules-core (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:17.558 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-fuse (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:17.786 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-07-26T06:02:17.786 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-07-26T06:02:18.234 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up libcephfs-dev (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:18.386 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-07-26T06:02:18.695 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-07-26T06:02:19.039 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-07-26T06:02:19.375 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rados (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:19.518 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rbd (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:19.669 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up rbd-fuse (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:19.795 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-routes (2.4.1-2) ... 2021-07-26T06:02:20.105 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-07-26T06:02:20.526 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-rgw (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:20.652 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-cephfs (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:20.795 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-07-26T06:02:21.715 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-common (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:22.323 INFO:teuthology.orchestra.run.smithi094.stdout:Adding group ceph....done 2021-07-26T06:02:22.742 INFO:teuthology.orchestra.run.smithi094.stdout:Adding system user ceph....done 2021-07-26T06:02:22.951 INFO:teuthology.orchestra.run.smithi094.stdout:Setting system user ceph properties....done 2021-07-26T06:02:22.976 INFO:teuthology.orchestra.run.smithi094.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-07-26T06:02:23.125 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-07-26T06:02:23.436 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-07-26T06:02:23.906 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-test (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:24.052 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up radosgw (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:24.355 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-07-26T06:02:24.356 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-07-26T06:02:25.013 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-base (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:25.378 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-07-26T06:02:25.825 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mds (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:26.055 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-07-26T06:02:26.055 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-07-26T06:02:26.493 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:26.733 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-07-26T06:02:26.733 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-07-26T06:02:27.201 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-osd (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:27.722 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-07-26T06:02:27.722 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-07-26T06:02:28.163 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-k8sevents (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:28.289 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:28.456 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mon (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:28.684 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-07-26T06:02:28.685 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-07-26T06:02:29.133 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-cephadm (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:29.309 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:29.435 INFO:teuthology.orchestra.run.smithi094.stdout:Setting up ceph-mgr-dashboard (17.0.0-6358-g197e564f-1focal) ... 2021-07-26T06:02:29.595 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-07-26T06:02:29.895 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-07-26T06:02:32.974 INFO:teuthology.orchestra.run.smithi094.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-07-26T06:02:33.404 INFO:teuthology.orchestra.run.smithi094.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-07-26T06:02:33.408 DEBUG:teuthology.parallel:result is None 2021-07-26T06:02:33.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo lsb_release -is 2021-07-26T06:02:33.452 INFO:teuthology.orchestra.run.smithi094.stdout:Ubuntu 2021-07-26T06:02:33.457 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-07-26T06:02:33.457 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:02:33.652 DEBUG:teuthology.orchestra.run.smithi094:> dpkg-query -W -f '${Version}' ceph 2021-07-26T06:02:33.674 INFO:teuthology.orchestra.run.smithi094.stdout:17.0.0-6358-g197e564f-1focal 2021-07-26T06:02:33.674 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-6358-g197e564f-1focal 2021-07-26T06:02:33.675 INFO:teuthology.task.install:The correct ceph version 17.0.0-6358-g197e564f-1focal is installed. 2021-07-26T06:02:33.675 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-07-26T06:02:33.676 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-07-26T06:02:33.676 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-07-26T06:02:33.692 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-07-26T06:02:33.693 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-07-26T06:02:33.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2021-07-26T06:02:33.706 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-07-26T06:02:33.764 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-07-26T06:02:33.765 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-07-26T06:02:33.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2021-07-26T06:02:33.775 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-07-26T06:02:33.786 INFO:teuthology.run_tasks:Running task workunit... 2021-07-26T06:02:33.803 INFO:tasks.workunit:Pulling workunits from ref 197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:02:33.804 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-07-26T06:02:33.805 INFO:tasks.workunit:timeout=3h 2021-07-26T06:02:33.805 INFO:tasks.workunit:cleanup=True 2021-07-26T06:02:33.805 DEBUG:teuthology.orchestra.run.smithi094:> stat -- /home/ubuntu/cephtest/mnt.0 2021-07-26T06:02:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-07-26T06:02:33.813 INFO:teuthology.orchestra.run.smithi094.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-07-26T06:02:33.813 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-07-26T06:02:33.867 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-07-26T06:02:33.868 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-07-26T06:02:33.875 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 197e564fdaee12596de1dc8f3253d1704ae66d5d 2021-07-26T06:02:33.883 INFO:tasks.workunit.client.0.smithi094.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-07-26T06:03:15.404 INFO:tasks.workunit.client.0.smithi094.stderr:Note: switching to '197e564fdaee12596de1dc8f3253d1704ae66d5d'. 2021-07-26T06:03:15.404 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:03:15.405 INFO:tasks.workunit.client.0.smithi094.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-07-26T06:03:15.405 INFO:tasks.workunit.client.0.smithi094.stderr:changes and commit them, and you can discard any commits you make in this 2021-07-26T06:03:15.405 INFO:tasks.workunit.client.0.smithi094.stderr:state without impacting any branches by switching back to a branch. 2021-07-26T06:03:15.405 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:03:15.406 INFO:tasks.workunit.client.0.smithi094.stderr:If you want to create a new branch to retain commits you create, you may 2021-07-26T06:03:15.406 INFO:tasks.workunit.client.0.smithi094.stderr:do so (now or later) by using -c with the switch command. Example: 2021-07-26T06:03:15.406 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:03:15.406 INFO:tasks.workunit.client.0.smithi094.stderr: git switch -c 2021-07-26T06:03:15.406 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:03:15.407 INFO:tasks.workunit.client.0.smithi094.stderr:Or undo this operation with: 2021-07-26T06:03:15.407 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:03:15.408 INFO:tasks.workunit.client.0.smithi094.stderr: git switch - 2021-07-26T06:03:15.408 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:03:15.408 INFO:tasks.workunit.client.0.smithi094.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-07-26T06:03:15.409 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:03:15.409 INFO:tasks.workunit.client.0.smithi094.stderr:HEAD is now at 197e564fdae Merge branch 'ups-eliminate-rollfwd' of https://github.com/LiumxNL/ceph into wip-kefu-testing-2021-07-26-1205 2021-07-26T06:03:15.411 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-07-26T06:03:15.421 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2021-07-26T06:03:15.422 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-07-26T06:03:15.475 INFO:tasks.workunit:Running workunits matching osd on client.0... 2021-07-26T06:03:15.475 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2021-07-26T06:03:15.476 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2021-07-26T06:03:15.502 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:03:15.543 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:03:15.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:03:15.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:03:15.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:03:15.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:03:15.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:03:15.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:03:15.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:03:15.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:03:15.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/bad-inc-map 2021-07-26T06:03:15.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2021-07-26T06:03:15.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2021-07-26T06:03:15.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:40391 2021-07-26T06:03:15.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:40391 2021-07-26T06:03:15.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2021-07-26T06:03:15.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2021-07-26T06:03:15.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none ' 2021-07-26T06:03:15.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:40391 ' 2021-07-26T06:03:15.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2021-07-26T06:03:15.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2021-07-26T06:03:15.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:03:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2021-07-26T06:03:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2021-07-26T06:03:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2021-07-26T06:03:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2021-07-26T06:03:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2021-07-26T06:03:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-07-26T06:03:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:03:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-07-26T06:03:15.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:03:15.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:03:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:03:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:03:15.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:03:15.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:03:15.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:03:15.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:03:15.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:03:15.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:03:15.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:03:15.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:03:15.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:03:15.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:03:15.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:03:15.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:03:15.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:03:15.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:03:15.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-07-26T06:03:15.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:03:15.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:15.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:15.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13570 2021-07-26T06:03:15.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:03:15.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:03:15.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2021-07-26T06:03:15.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:03:15.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:15.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:15.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.13570 2021-07-26T06:03:15.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:03:15.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:03:15.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:03:15.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:03:15.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2021-07-26T06:03:15.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2021-07-26T06:03:15.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2021-07-26T06:03:15.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2021-07-26T06:03:15.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2021-07-26T06:03:15.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:03:15.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:03:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:03:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2021-07-26T06:03:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2021-07-26T06:03:16.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:03:16.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:03:16.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:03:16.145 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:03:16.146 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:16.146 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:16.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:16.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:03:16.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:03:16.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:03:16.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:03:16.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:03:16.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:03:16.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:03:16.430 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:03:16.431 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:03:16.431 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:03:16.431 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:03:16.431 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:16.432 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:16.432 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13570/ceph-mon.a.asok 2021-07-26T06:03:16.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:03:16.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.13570/ceph-mon.a.asok config get fsid 2021-07-26T06:03:16.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:03:16.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:03:16.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:03:16.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:03:16.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:03:16.543 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:03:16.544 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:03:16.544 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:03:16.544 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:03:16.544 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:16.545 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:16.545 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.13570/ceph-mon.a.asok 2021-07-26T06:03:16.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:03:16.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.13570/ceph-mon.a.asok config get mon_host 2021-07-26T06:03:16.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2021-07-26T06:03:16.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2021-07-26T06:03:16.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:03:16.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:03:16.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:03:16.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2021-07-26T06:03:16.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:03:17.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:03:17.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:03:17.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:03:17.507 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:03:17.507 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:17.508 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:17.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:17.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:03:17.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:03:17.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2021-07-26T06:03:17.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-07-26T06:03:17.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:03:17.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:03:17.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:03:17.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2021-07-26T06:03:17.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 ' 2021-07-26T06:03:17.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:03:17.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:03:17.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:03:17.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2021-07-26T06:03:17.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2021-07-26T06:03:17.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:03:17.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:03:17.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:03:17.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-07-26T06:03:17.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:03:17.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:03:17.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:03:17.587 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:03:17.588 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:17.588 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:17.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:17.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:17.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:03:17.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:03:17.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:03:17.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-07-26T06:03:17.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-07-26T06:03:17.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:03:17.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:03:17.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:03:17.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:03:17.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:03:17.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2021-07-26T06:03:17.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:03:17.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1eca21ca-a2d3-48f5-a202-01fbb2002dd3 2021-07-26T06:03:17.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1eca21ca-a2d3-48f5-a202-01fbb2002dd3' 2021-07-26T06:03:17.592 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 1eca21ca-a2d3-48f5-a202-01fbb2002dd3 2021-07-26T06:03:17.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:03:17.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlUP5g4XIPJBAAQqg80TEBGpXlOeot31ngOw== 2021-07-26T06:03:17.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlUP5g4XIPJBAAQqg80TEBGpXlOeot31ngOw=="}' 2021-07-26T06:03:17.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1eca21ca-a2d3-48f5-a202-01fbb2002dd3 -i td/bad-inc-map/0/new.json 2021-07-26T06:03:17.980 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:03:17.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2021-07-26T06:03:17.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAlUP5g4XIPJBAAQqg80TEBGpXlOeot31ngOw== --osd-uuid 1eca21ca-a2d3-48f5-a202-01fbb2002dd3 2021-07-26T06:03:18.042 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:18.040+0000 7fcc5b146f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:18.079 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:18.076+0000 7fcc5b146f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:18.081 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:18.080+0000 7fcc5b146f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:18.082 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:18.080+0000 7fcc5b146f40 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-07-26T06:03:18.082 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:18.080+0000 7fcc5b146f40 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-07-26T06:03:18.082 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:18.080+0000 7fcc5b146f40 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-07-26T06:03:18.126 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:18.124+0000 7fcc5b146f40 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2021-07-26T06:03:24.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2021-07-26T06:03:24.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:03:24.029 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:03:24.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:03:24.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:03:24.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:03:24.253 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:03:24.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:03:24.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:03:24.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:03:24.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:03:24.277 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:24.276+0000 7efc8058ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:24.283 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:24.280+0000 7efc8058ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:24.285 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:24.284+0000 7efc8058ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:24.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:03:24.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:03:24.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:03:24.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:03:24.517 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:03:24.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:03:24.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:24.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:03:24.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:24.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:03:24.583 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:24.580+0000 7efc8058ff40 -1 Falling back to public interface 2021-07-26T06:03:24.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:25.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:25.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:25.841 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:03:25.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:03:25.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:25.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:03:26.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:26.988 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:26.984+0000 7efc8058ff40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:03:27.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:27.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:27.209 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:03:27.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:03:27.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:27.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:03:27.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:28.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:28.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:28.634 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:03:28.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:03:28.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:28.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:03:29.001 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/14654,v1:127.0.0.1:6803/14654] [v2:127.0.0.1:6804/14654,v1:127.0.0.1:6805/14654] exists,up 1eca21ca-a2d3-48f5-a202-01fbb2002dd3 2021-07-26T06:03:29.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:03:29.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:03:29.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:03:29.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2021-07-26T06:03:29.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-07-26T06:03:29.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:03:29.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:03:29.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:03:29.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2021-07-26T06:03:29.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 ' 2021-07-26T06:03:29.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:03:29.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:03:29.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:03:29.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2021-07-26T06:03:29.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2021-07-26T06:03:29.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:03:29.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:03:29.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:03:29.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-07-26T06:03:29.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:03:29.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:03:29.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:03:29.011 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:03:29.011 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:29.011 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:29.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:29.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:29.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:03:29.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:03:29.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:03:29.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-07-26T06:03:29.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-07-26T06:03:29.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:03:29.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:03:29.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:03:29.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:03:29.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:03:29.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2021-07-26T06:03:29.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:03:29.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=410af6f4-5bb3-41ce-b3ee-cfabd153d41b 2021-07-26T06:03:29.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 410af6f4-5bb3-41ce-b3ee-cfabd153d41b' 2021-07-26T06:03:29.015 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 410af6f4-5bb3-41ce-b3ee-cfabd153d41b 2021-07-26T06:03:29.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:03:29.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxUP5g4U6GARAAPKqZlP75Vjt39zZeglY6XA== 2021-07-26T06:03:29.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxUP5g4U6GARAAPKqZlP75Vjt39zZeglY6XA=="}' 2021-07-26T06:03:29.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 410af6f4-5bb3-41ce-b3ee-cfabd153d41b -i td/bad-inc-map/1/new.json 2021-07-26T06:03:29.566 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:03:29.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2021-07-26T06:03:29.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAxUP5g4U6GARAAPKqZlP75Vjt39zZeglY6XA== --osd-uuid 410af6f4-5bb3-41ce-b3ee-cfabd153d41b 2021-07-26T06:03:29.611 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:29.608+0000 7fdf331dbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:29.615 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:29.612+0000 7fdf331dbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:29.616 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:29.616+0000 7fdf331dbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:29.619 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:29.616+0000 7fdf331dbf40 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-07-26T06:03:29.619 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:29.616+0000 7fdf331dbf40 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-07-26T06:03:29.619 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:29.616+0000 7fdf331dbf40 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-07-26T06:03:29.702 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:29.704+0000 7fdf331dbf40 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2021-07-26T06:03:35.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2021-07-26T06:03:35.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:03:35.877 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:03:35.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:03:35.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:03:36.385 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:03:36.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:03:36.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:03:36.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:03:36.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:03:36.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:03:36.412 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:36.412+0000 7fa55dd71f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:36.429 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:36.428+0000 7fa55dd71f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:36.430 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:36.428+0000 7fa55dd71f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:36.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:03:36.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:03:36.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:03:36.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:03:36.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:03:36.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:36.774 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:03:36.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:03:36.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:36.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:03:36.979 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:36.972+0000 7fa55dd71f40 -1 Falling back to public interface 2021-07-26T06:03:37.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:38.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:38.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:38.200 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:03:38.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:03:38.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:38.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:03:38.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:39.282 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:39.280+0000 7fa55dd71f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:03:39.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:39.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:39.546 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:03:39.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:03:39.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:39.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:03:39.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:40.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:40.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:40.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:03:40.994 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:03:40.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:40.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:03:41.349 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/16084,v1:127.0.0.1:6811/16084] [v2:127.0.0.1:6812/16084,v1:127.0.0.1:6813/16084] exists,up 410af6f4-5bb3-41ce-b3ee-cfabd153d41b 2021-07-26T06:03:41.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:03:41.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:03:41.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:03:41.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2021-07-26T06:03:41.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2021-07-26T06:03:41.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:03:41.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:03:41.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:03:41.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2021-07-26T06:03:41.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 ' 2021-07-26T06:03:41.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:03:41.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:03:41.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:03:41.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2021-07-26T06:03:41.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2021-07-26T06:03:41.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:03:41.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:03:41.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:03:41.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-07-26T06:03:41.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:03:41.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:03:41.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:03:41.357 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:03:41.358 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:03:41.358 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:03:41.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:41.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' 2021-07-26T06:03:41.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:03:41.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:03:41.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:03:41.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-07-26T06:03:41.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-07-26T06:03:41.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:03:41.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:03:41.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:03:41.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:03:41.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:03:41.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2021-07-26T06:03:41.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:03:41.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1070b173-858c-403e-a95b-84b675a8b214 2021-07-26T06:03:41.363 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 1070b173-858c-403e-a95b-84b675a8b214 2021-07-26T06:03:41.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1070b173-858c-403e-a95b-84b675a8b214' 2021-07-26T06:03:41.364 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:03:41.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9UP5gmhy1FhAAEQDMMK6oqby3HQZE5j5ocg== 2021-07-26T06:03:41.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9UP5gmhy1FhAAEQDMMK6oqby3HQZE5j5ocg=="}' 2021-07-26T06:03:41.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1070b173-858c-403e-a95b-84b675a8b214 -i td/bad-inc-map/2/new.json 2021-07-26T06:03:41.933 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:03:41.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2021-07-26T06:03:41.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA9UP5gmhy1FhAAEQDMMK6oqby3HQZE5j5ocg== --osd-uuid 1070b173-858c-403e-a95b-84b675a8b214 2021-07-26T06:03:41.972 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:41.968+0000 7f5980045f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:41.974 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:41.972+0000 7f5980045f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:41.976 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:41.976+0000 7f5980045f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:41.977 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:41.976+0000 7f5980045f40 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-07-26T06:03:41.977 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:41.976+0000 7f5980045f40 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-07-26T06:03:41.977 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:41.976+0000 7f5980045f40 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-07-26T06:03:42.088 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:42.088+0000 7f5980045f40 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2021-07-26T06:03:50.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2021-07-26T06:03:50.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:03:50.987 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:03:50.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:03:50.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:03:51.471 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:03:51.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:03:51.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=968b44bf-daaa-426a-9419-7b9e16053738 --auth-supported=none --mon-host=127.0.0.1:40391 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.13570/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:03:51.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:03:51.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:03:51.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:03:51.498 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:51.496+0000 7fb7e4196f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:51.500 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:51.500+0000 7fb7e4196f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:51.504 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:51.500+0000 7fb7e4196f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:03:51.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:03:51.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:03:51.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:03:51.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:03:51.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:03:51.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:51.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:03:51.896 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:03:51.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:51.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:03:52.069 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:52.068+0000 7fb7e4196f40 -1 Falling back to public interface 2021-07-26T06:03:52.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:53.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:53.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:53.303 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:03:53.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:03:53.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:53.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:03:53.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:54.609 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:03:54.608+0000 7fb7e4196f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:03:54.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:54.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:54.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:03:54.650 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:03:54.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:54.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:03:55.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:03:56.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:03:56.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:03:56.013 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:03:56.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:03:56.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:03:56.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:03:56.425 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/17490,v1:127.0.0.1:6819/17490] [v2:127.0.0.1:6820/17490,v1:127.0.0.1:6821/17490] exists,up 1070b173-858c-403e-a95b-84b675a8b214 2021-07-26T06:03:56.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:03:56.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:03:56.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:03:56.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2021-07-26T06:03:56.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2021-07-26T06:03:56.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2021-07-26T06:03:57.544 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'foo' already exists 2021-07-26T06:03:57.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:03:58.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2021-07-26T06:03:59.864 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 min_size to 1 2021-07-26T06:03:59.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2021-07-26T06:04:01.153 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 min_size to 2 2021-07-26T06:04:01.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2021-07-26T06:04:06.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2021-07-26T06:04:06.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2021-07-26T06:04:06.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:04:06.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:04:06.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:04:06.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:04:06.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 10 )) 2021-07-26T06:04:06.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:04:06.167 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:04:06.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:06.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:04:06.551 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/14654,v1:127.0.0.1:6803/14654] [v2:127.0.0.1:6804/14654,v1:127.0.0.1:6805/14654] exists,up 1eca21ca-a2d3-48f5-a202-01fbb2002dd3 2021-07-26T06:04:06.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:04:06.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:04:06.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:04:06.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2021-07-26T06:04:06.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2021-07-26T06:04:06.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:04:06.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:04:06.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:04:06.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:04:06.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 10 )) 2021-07-26T06:04:06.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:04:06.555 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:04:06.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:06.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:04:06.945 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/16084,v1:127.0.0.1:6811/16084] [v2:127.0.0.1:6812/16084,v1:127.0.0.1:6813/16084] exists,up 410af6f4-5bb3-41ce-b3ee-cfabd153d41b 2021-07-26T06:04:06.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:04:06.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:04:06.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:04:06.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2021-07-26T06:04:06.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2021-07-26T06:04:06.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:04:06.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:04:06.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:04:06.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:04:06.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 10 )) 2021-07-26T06:04:06.949 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:04:06.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:04:06.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:06.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:04:07.309 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/17490,v1:127.0.0.1:6819/17490] [v2:127.0.0.1:6820/17490,v1:127.0.0.1:6821/17490] exists,up 1070b173-858c-403e-a95b-84b675a8b214 2021-07-26T06:04:07.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:04:07.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:04:07.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:04:07.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2021-07-26T06:04:07.314 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:57.392+0000 7fb7d7969700 -1 osd.2 15 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.314 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:57.696+0000 7fb7d7969700 -1 osd.2 17 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.315 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:58.060+0000 7fb7d7969700 -1 osd.2 18 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.315 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:58.712+0000 7fb7d7969700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.315 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:59.564+0000 7fb7d5965700 -1 osd.2 20 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.315 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.024+0000 7fb7d5965700 -1 osd.2 23 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.315 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.280+0000 7fb7d7969700 -1 osd.2 24 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.316 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.284+0000 7fb7d7969700 -1 osd.2 24 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.316 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.724+0000 7fb7d7969700 -1 osd.2 25 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.316 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.340+0000 7fb7d5965700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.316 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.340+0000 7fb7d5965700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.317 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.356+0000 7fb7d5965700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.317 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.356+0000 7fb7d5965700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.317 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.500+0000 7fb7d7969700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.317 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.912+0000 7fb7d7969700 -1 osd.2 28 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.317 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.912+0000 7fb7d7969700 -1 osd.2 28 handle_osd_map injecting map crc failure 2021-07-26T06:04:07.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2021-07-26T06:04:07.319 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:57.392+0000 7fb7d7969700 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2021-07-26T06:04:07.320 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:57.696+0000 7fb7d7969700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2021-07-26T06:04:07.320 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:58.060+0000 7fb7d7969700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2021-07-26T06:04:07.320 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:58.712+0000 7fb7d7969700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-07-26T06:04:07.320 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:03:59.564+0000 7fb7d5965700 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2021-07-26T06:04:07.321 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.024+0000 7fb7d5965700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2021-07-26T06:04:07.321 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.280+0000 7fb7d7969700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2021-07-26T06:04:07.321 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.284+0000 7fb7d7969700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2021-07-26T06:04:07.321 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:00.728+0000 7fb7d7969700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2021-07-26T06:04:07.321 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.340+0000 7fb7d5965700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-07-26T06:04:07.322 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.340+0000 7fb7d5965700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-07-26T06:04:07.322 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.356+0000 7fb7d5965700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-07-26T06:04:07.322 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.356+0000 7fb7d5965700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-07-26T06:04:07.322 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.500+0000 7fb7d7969700 10 osd.2 26 handle_osd_map bailing because last < start (27<28) 2021-07-26T06:04:07.323 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.912+0000 7fb7d7969700 10 osd.2 28 handle_osd_map bailing because last < start (28<29) 2021-07-26T06:04:07.323 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:04:01.916+0000 7fb7d7969700 10 osd.2 28 handle_osd_map bailing because last < start (28<29) 2021-07-26T06:04:07.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2021-07-26T06:04:07.324 INFO:tasks.workunit.client.0.smithi094.stdout:success 2021-07-26T06:04:07.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2021-07-26T06:04:07.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2021-07-26T06:04:07.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2021-07-26T06:04:07.939 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'foo' does not exist 2021-07-26T06:04:07.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2021-07-26T06:04:07.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:04:07.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:04:07.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:04:07.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:04:07.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:04:08.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:04:08.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2021-07-26T06:04:08.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-07-26T06:04:08.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:04:08.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-07-26T06:04:08.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:04:08.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:04:08.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:04:08.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:04:08.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:04:08.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:04:08.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:04:08.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:04:08.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:04:08.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:04:08.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:04:08.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:04:08.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:04:08.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:04:08.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:04:08.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:04:08.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:04:08.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:04:08.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-07-26T06:04:08.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:04:08.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:08.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:04:08.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13570 2021-07-26T06:04:08.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:04:08.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:04:08.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:04:08.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/bad-inc-map 0 2021-07-26T06:04:08.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-07-26T06:04:08.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:04:08.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-07-26T06:04:08.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:04:08.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:04:08.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:04:08.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:04:08.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:04:08.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:04:08.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:04:08.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:04:08.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:04:08.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:04:08.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:04:08.108 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:04:08.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:04:08.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:04:08.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:04:08.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:04:08.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:04:08.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:04:08.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-07-26T06:04:08.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:04:08.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:08.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.13570 2021-07-26T06:04:08.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.13570 2021-07-26T06:04:08.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:04:08.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:04:08.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:04:08.119 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:04:08.119 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:04:08.134 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2021-07-26T06:04:08.135 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2021-07-26T06:04:08.196 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:04:08.209 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:04:08.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:04:08.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:04:08.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:04:08.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:04:08.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:04:08.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:04:08.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:04:08.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:04:08.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/divergent-priors 2021-07-26T06:04:08.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2021-07-26T06:04:08.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2021-07-26T06:04:08.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2021-07-26T06:04:08.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2021-07-26T06:04:08.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2021-07-26T06:04:08.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2021-07-26T06:04:08.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2021-07-26T06:04:08.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2021-07-26T06:04:08.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2021-07-26T06:04:08.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2021-07-26T06:04:08.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2021-07-26T06:04:08.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2021-07-26T06:04:08.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2021-07-26T06:04:08.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2021-07-26T06:04:08.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2021-07-26T06:04:08.217 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2021-07-26T06:04:08.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2021-07-26T06:04:08.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2021-07-26T06:04:08.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2021-07-26T06:04:08.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2021-07-26T06:04:08.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2021-07-26T06:04:08.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2021-07-26T06:04:08.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2021-07-26T06:04:08.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2021-07-26T06:04:08.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2021-07-26T06:04:08.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2021-07-26T06:04:08.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none ' 2021-07-26T06:04:08.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2021-07-26T06:04:08.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T06:04:08.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2021-07-26T06:04:08.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:04:08.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2021-07-26T06:04:08.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:04:08.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2021-07-26T06:04:08.225 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_divergent_2 2021-07-26T06:04:08.225 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_divergent_3 2021-07-26T06:04:08.225 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_divergent_ec' 2021-07-26T06:04:08.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-07-26T06:04:08.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-07-26T06:04:08.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-07-26T06:04:08.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-07-26T06:04:08.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:04:08.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:04:08.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:04:08.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:04:08.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:04:08.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:04:08.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:04:08.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:04:08.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:04:08.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:04:08.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:04:08.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:04:08.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:04:08.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:04:08.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:04:08.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:04:08.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:04:08.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:04:08.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:04:08.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:04:08.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:04:08.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:04:08.242 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:04:08.242 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:08.242 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:08.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:04:08.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:04:08.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:04:08.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-07-26T06:04:08.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:04:08.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:08.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:08.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.18330 2021-07-26T06:04:08.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:04:08.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:04:08.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:04:08.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:04:08.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-07-26T06:04:08.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2021-07-26T06:04:08.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2021-07-26T06:04:08.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2021-07-26T06:04:08.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2021-07-26T06:04:08.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2021-07-26T06:04:08.252 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2021-07-26T06:04:08.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2021-07-26T06:04:08.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2021-07-26T06:04:08.271 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:04:08.271 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:04:08.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2021-07-26T06:04:08.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-07-26T06:04:08.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:04:08.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:04:08.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:04:08.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-07-26T06:04:08.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-07-26T06:04:08.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:04:08.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:04:08.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:04:08.676 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:04:08.676 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:08.676 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:08.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:08.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:04:08.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:04:08.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:04:08.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:04:08.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:04:08.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:04:08.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:04:08.916 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:04:08.917 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:04:08.917 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:04:08.917 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:04:08.917 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:08.918 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:08.918 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:04:08.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:04:08.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get fsid 2021-07-26T06:04:09.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:04:09.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:04:09.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:04:09.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:04:09.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:04:09.012 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:04:09.013 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:04:09.013 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:04:09.013 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:04:09.013 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:09.013 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:09.014 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:04:09.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:04:09.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get mon_host 2021-07-26T06:04:09.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2021-07-26T06:04:09.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-07-26T06:04:09.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:04:09.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:04:09.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:04:09.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-07-26T06:04:09.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:04:10.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:04:10.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:04:10.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:04:10.021 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:04:10.022 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:10.022 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:10.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:10.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:04:10.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:04:10.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-07-26T06:04:10.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2021-07-26T06:04:10.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:04:10.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:04:10.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:04:10.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:04:10.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:04:10.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:04:10.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:04:10.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:04:10.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:04:10.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:04:10.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:04:10.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:04:10.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:04:10.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:04:10.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:04:10.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:04:10.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:04:10.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:04:10.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:04:10.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:10.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:10.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:04:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:04:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:04:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:04:10.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:04:10.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:04:10.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:04:10.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:04:10.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:04:10.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:04:10.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:04:10.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:04:10.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=827849e5-5a86-4978-bf98-6b5fbb0f3a6b 2021-07-26T06:04:10.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 827849e5-5a86-4978-bf98-6b5fbb0f3a6b' 2021-07-26T06:04:10.073 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 827849e5-5a86-4978-bf98-6b5fbb0f3a6b 2021-07-26T06:04:10.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:04:10.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaUP5gH6jBBRAAHtb4J5kmWdLqJZZkOtKC5A== 2021-07-26T06:04:10.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaUP5gH6jBBRAAHtb4J5kmWdLqJZZkOtKC5A=="}' 2021-07-26T06:04:10.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 827849e5-5a86-4978-bf98-6b5fbb0f3a6b -i td/divergent-priors/0/new.json 2021-07-26T06:04:10.457 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:04:10.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2021-07-26T06:04:10.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBaUP5gH6jBBRAAHtb4J5kmWdLqJZZkOtKC5A== --osd-uuid 827849e5-5a86-4978-bf98-6b5fbb0f3a6b 2021-07-26T06:04:10.506 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:10.511+0000 7f16b6e1ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:10.509 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:10.515+0000 7f16b6e1ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:10.511 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:10.515+0000 7f16b6e1ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:10.512 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:10.515+0000 7f16b6e1ef40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:04:10.512 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:10.515+0000 7f16b6e1ef40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:04:10.512 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:10.515+0000 7f16b6e1ef40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:04:10.581 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:10.587+0000 7f16b6e1ef40 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-07-26T06:04:16.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-07-26T06:04:16.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:04:16.244 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:04:16.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:04:16.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:04:16.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:04:16.764 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:04:16.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:04:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:04:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:04:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:04:16.800 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:16.805+0000 7f15f466ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:16.833+0000 7f15f466ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:16.833+0000 7f15f466ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:17.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:04:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:04:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:04:17.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:04:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:04:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:17.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:04:17.145 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:04:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:04:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:17.381+0000 7f15f466ff40 -1 Falling back to public interface 2021-07-26T06:04:17.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:18.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:18.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:18.507 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:04:18.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:04:18.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:18.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:04:18.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:19.662 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:19.665+0000 7f15f466ff40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:04:19.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:19.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:19.884 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:04:19.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:04:19.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:19.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:04:20.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:21.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:21.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:21.392 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:04:21.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:04:21.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:21.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:04:21.770 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/19441,v1:127.0.0.1:6803/19441] [v2:127.0.0.1:6804/19441,v1:127.0.0.1:6805/19441] exists,up 827849e5-5a86-4978-bf98-6b5fbb0f3a6b 2021-07-26T06:04:21.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:04:21.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:04:21.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:04:21.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-07-26T06:04:21.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2021-07-26T06:04:21.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:04:21.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:04:21.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:04:21.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:04:21.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:04:21.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:04:21.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:04:21.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:04:21.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:04:21.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:04:21.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:04:21.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:04:21.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:04:21.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:04:21.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:04:21.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:04:21.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:04:21.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:04:21.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:04:21.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:21.781 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:21.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:21.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:21.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:04:21.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:04:21.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:04:21.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:04:21.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:04:21.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:04:21.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:04:21.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:04:21.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:04:21.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:04:21.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:04:21.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:04:21.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1 2021-07-26T06:04:21.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1' 2021-07-26T06:04:21.784 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1 2021-07-26T06:04:21.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:04:21.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlUP5gULosMBAAjWs892T+Zp4xq7yYTlXlqg== 2021-07-26T06:04:21.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlUP5gULosMBAAjWs892T+Zp4xq7yYTlXlqg=="}' 2021-07-26T06:04:21.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1 -i td/divergent-priors/1/new.json 2021-07-26T06:04:22.282 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:04:22.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2021-07-26T06:04:22.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBlUP5gULosMBAAjWs892T+Zp4xq7yYTlXlqg== --osd-uuid e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1 2021-07-26T06:04:22.334 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:22.337+0000 7f8e81e2af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:22.337 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:22.341+0000 7f8e81e2af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:22.339 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:22.345+0000 7f8e81e2af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:22.340 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:22.345+0000 7f8e81e2af40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:04:22.340 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:22.345+0000 7f8e81e2af40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:04:22.340 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:22.345+0000 7f8e81e2af40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:04:22.426 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:22.429+0000 7f8e81e2af40 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-07-26T06:04:28.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-07-26T06:04:28.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:04:28.237 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:04:28.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:04:28.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:04:28.738 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:04:28.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:04:28.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:04:28.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:04:28.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:04:28.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:04:28.775 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:28.781+0000 7ff32010ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:28.835 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:28.781+0000 7ff32010ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:28.835 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:28.785+0000 7ff32010ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:29.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:04:29.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:04:29.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:04:29.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:04:29.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:04:29.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:29.070 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:04:29.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:04:29.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:29.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:04:29.323 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:29.329+0000 7ff32010ff40 -1 Falling back to public interface 2021-07-26T06:04:29.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:30.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:30.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:30.504 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:04:30.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:04:30.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:30.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:04:30.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:31.536 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:31.541+0000 7ff32010ff40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:04:31.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:31.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:31.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:04:31.880 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:04:31.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:31.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:04:32.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:33.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:33.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:33.313 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:04:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:04:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:04:33.711 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/20846,v1:127.0.0.1:6811/20846] [v2:127.0.0.1:6812/20846,v1:127.0.0.1:6813/20846] exists,up e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1 2021-07-26T06:04:33.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:04:33.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:04:33.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:04:33.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-07-26T06:04:33.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2021-07-26T06:04:33.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:04:33.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:04:33.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:04:33.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:04:33.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:04:33.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:04:33.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:04:33.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:04:33.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:04:33.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:04:33.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:04:33.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:04:33.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:04:33.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:04:33.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:04:33.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:04:33.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:04:33.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:04:33.719 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:04:33.719 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:04:33.719 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:04:33.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:33.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:04:33.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:04:33.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:04:33.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:04:33.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:04:33.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:04:33.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:04:33.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:04:33.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:04:33.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:04:33.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:04:33.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:04:33.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:04:33.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1017022a-e6dc-4403-80b8-d7ce70ebba56 2021-07-26T06:04:33.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1017022a-e6dc-4403-80b8-d7ce70ebba56' 2021-07-26T06:04:33.725 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 1017022a-e6dc-4403-80b8-d7ce70ebba56 2021-07-26T06:04:33.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:04:33.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxUP5gfpDPLBAAzNpTa0Zbu3kuiMbkR+tYZQ== 2021-07-26T06:04:33.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxUP5gfpDPLBAAzNpTa0Zbu3kuiMbkR+tYZQ=="}' 2021-07-26T06:04:33.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1017022a-e6dc-4403-80b8-d7ce70ebba56 -i td/divergent-priors/2/new.json 2021-07-26T06:04:34.233 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:04:34.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2021-07-26T06:04:34.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBxUP5gfpDPLBAAzNpTa0Zbu3kuiMbkR+tYZQ== --osd-uuid 1017022a-e6dc-4403-80b8-d7ce70ebba56 2021-07-26T06:04:34.286 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:34.293+0000 7ff9810f8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:34.355 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:34.361+0000 7ff9810f8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:34.357 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:34.361+0000 7ff9810f8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:34.358 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:34.365+0000 7ff9810f8f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:04:34.358 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:34.365+0000 7ff9810f8f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:04:34.359 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:34.365+0000 7ff9810f8f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:04:34.459 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:34.465+0000 7ff9810f8f40 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-07-26T06:04:40.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-07-26T06:04:40.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:04:40.480 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:04:40.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:04:40.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:04:40.957 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:04:40.957 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:04:40.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:04:40.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:04:40.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:04:40.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:04:40.994 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:41.001+0000 7fdfe9d30f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:40.996 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:41.001+0000 7fdfe9d30f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:40.997 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:41.001+0000 7fdfe9d30f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:04:41.284 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:41.289+0000 7fdfe9d30f40 -1 Falling back to public interface 2021-07-26T06:04:41.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:04:41.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:04:41.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:04:41.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:04:41.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:04:41.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:41.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:04:41.336 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:04:41.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:41.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:04:41.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:42.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:42.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:42.707 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:04:42.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:04:42.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:42.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:04:43.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:43.814 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:04:43.821+0000 7fdfe9d30f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:04:44.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:44.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:44.082 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:04:44.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:04:44.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:44.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:04:44.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:04:45.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:04:45.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:04:45.540 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:04:45.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:04:45.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:04:45.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:04:45.970 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/22252,v1:127.0.0.1:6819/22252] [v2:127.0.0.1:6820/22252,v1:127.0.0.1:6821/22252] exists,up 1017022a-e6dc-4403-80b8-d7ce70ebba56 2021-07-26T06:04:45.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:04:45.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:04:45.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:04:45.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2021-07-26T06:04:46.874 INFO:tasks.workunit.client.0.smithi094.stderr:noout is set 2021-07-26T06:04:46.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2021-07-26T06:04:47.760 INFO:tasks.workunit.client.0.smithi094.stderr:noin is set 2021-07-26T06:04:47.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2021-07-26T06:04:48.717 INFO:tasks.workunit.client.0.smithi094.stderr:nodown is set 2021-07-26T06:04:48.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2021-07-26T06:04:48.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T06:04:49.417 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T06:04:49.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:04:50.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2021-07-26T06:04:51.286 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 3 2021-07-26T06:04:51.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2021-07-26T06:04:52.099 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 min_size to 2 2021-07-26T06:04:52.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2021-07-26T06:04:52.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:04:52.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:04:52.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:04:52.616 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:04:52.616 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:04:52.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:04:52.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:04:52.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:04:52.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T06:04:52.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T06:04:52.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T06:04:52.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:04:52.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:04:52.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:04:52.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:04:52.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2021-07-26T06:04:52.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:04:52.937 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:04:53.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509444 2021-07-26T06:04:53.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509444 2021-07-26T06:04:53.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2021-07-26T06:04:53.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:04:53.091 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T06:04:53.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:04:53.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:04:53.095 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T06:04:53.095 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:04:53.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T06:04:53.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T06:04:53.097 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836489 2021-07-26T06:04:53.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:04:53.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836489 2021-07-26T06:04:53.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:04:54.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:04:54.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:04:54.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T06:04:54.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:04:54.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:04:54.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:04:54.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:04:54.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:04:54.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:04:54.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:04:54.861 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672966 2021-07-26T06:04:54.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:04:54.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:04:55.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672966 2021-07-26T06:04:55.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:04:55.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509444 2021-07-26T06:04:55.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:04:55.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:04:55.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509444 2021-07-26T06:04:55.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:04:55.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509444 2021-07-26T06:04:55.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-07-26T06:04:55.260 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509444 2021-07-26T06:04:55.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:04:55.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509444 -lt 64424509444 2021-07-26T06:04:55.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2021-07-26T06:04:55.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:04:55.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:04:55.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:04:55.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:04:55.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:04:55.655 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:04:55.656 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:04:55.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:04:55.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:04:55.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:04:55.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:04:55.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:04:55.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:04:55.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:04:55.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:04:56.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:04:56.219 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:04:56.219 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:04:56.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:04:56.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:04:56.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:04:56.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836492 2021-07-26T06:04:56.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836492 2021-07-26T06:04:56.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492' 2021-07-26T06:04:56.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:04:56.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:04:56.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672969 2021-07-26T06:04:56.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672969 2021-07-26T06:04:56.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2021-07-26T06:04:56.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:04:56.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:04:56.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509447 2021-07-26T06:04:56.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509447 2021-07-26T06:04:56.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492 1-42949672969 2-64424509447' 2021-07-26T06:04:56.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:04:56.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836492 2021-07-26T06:04:56.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:04:56.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:04:56.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836492 2021-07-26T06:04:56.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:04:56.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836492 2021-07-26T06:04:56.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-07-26T06:04:56.733 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836492 2021-07-26T06:04:56.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:04:57.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836492 2021-07-26T06:04:57.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:04:58.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:04:58.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:04:58.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836492 -lt 21474836492 2021-07-26T06:04:58.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:04:58.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672969 2021-07-26T06:04:58.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:04:58.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:04:58.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672969 2021-07-26T06:04:58.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:04:58.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672969 2021-07-26T06:04:58.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-07-26T06:04:58.482 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672969 2021-07-26T06:04:58.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:04:58.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672969 2021-07-26T06:04:58.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:04:58.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509447 2021-07-26T06:04:58.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:04:58.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:04:58.875 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509447 2021-07-26T06:04:58.875 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:04:58.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509447 2021-07-26T06:04:58.877 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509447 2021-07-26T06:04:58.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-07-26T06:04:58.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:04:59.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509447 -lt 64424509447 2021-07-26T06:04:59.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:04:59.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:04:59.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:04:59.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:04:59.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:04:59.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:04:59.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:04:59.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:04:59.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:04:59.705 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:04:59.705 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:05:00.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:05:00.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:05:00.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:00.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:00.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:05:00.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:05:00.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:05:00.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2021-07-26T06:05:00.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2021-07-26T06:05:01.006 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:05:01.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2021-07-26T06:05:01.015 INFO:tasks.workunit.client.0.smithi094.stdout:primary and soon to be divergent is 1 2021-07-26T06:05:01.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2021-07-26T06:05:01.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2021-07-26T06:05:01.382 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:05:01.382 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:04:51.154745+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:05:01.382 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:05:01.383 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:05:01.384 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:05:01.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2021-07-26T06:05:01.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-07-26T06:05:01.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2021-07-26T06:05:01.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2021-07-26T06:05:01.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-07-26T06:05:01.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2021-07-26T06:05:01.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2021-07-26T06:05:01.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-07-26T06:05:01.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2021-07-26T06:05:01.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2021-07-26T06:05:01.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2021-07-26T06:05:01.396 INFO:tasks.workunit.client.0.smithi094.stdout:writing initial objects 2021-07-26T06:05:01.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2021-07-26T06:05:01.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:01.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2021-07-26T06:05:01.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:01.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2021-07-26T06:05:01.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:01.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2021-07-26T06:05:01.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:01.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2021-07-26T06:05:02.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:02.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2021-07-26T06:05:02.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:02.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2021-07-26T06:05:02.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:02.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2021-07-26T06:05:02.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:02.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2021-07-26T06:05:02.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:02.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2021-07-26T06:05:02.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:02.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2021-07-26T06:05:02.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:02.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2021-07-26T06:05:03.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:03.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2021-07-26T06:05:03.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:03.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2021-07-26T06:05:03.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:03.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2021-07-26T06:05:03.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:03.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2021-07-26T06:05:03.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:03.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2021-07-26T06:05:03.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:03.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2021-07-26T06:05:03.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:03.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2021-07-26T06:05:04.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:04.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2021-07-26T06:05:04.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:04.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2021-07-26T06:05:04.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:04.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2021-07-26T06:05:04.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:04.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2021-07-26T06:05:04.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:04.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2021-07-26T06:05:04.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:04.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2021-07-26T06:05:05.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:05.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2021-07-26T06:05:05.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:05.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2021-07-26T06:05:05.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:05.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2021-07-26T06:05:05.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:05.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2021-07-26T06:05:05.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:05.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2021-07-26T06:05:05.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:05.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2021-07-26T06:05:05.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:05.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2021-07-26T06:05:06.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:06.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2021-07-26T06:05:06.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:06.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2021-07-26T06:05:06.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:06.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2021-07-26T06:05:06.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:06.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2021-07-26T06:05:06.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:06.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2021-07-26T06:05:06.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:06.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2021-07-26T06:05:06.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:06.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2021-07-26T06:05:07.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:07.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2021-07-26T06:05:07.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:07.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2021-07-26T06:05:07.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:07.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2021-07-26T06:05:07.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:07.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2021-07-26T06:05:07.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:07.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2021-07-26T06:05:07.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:07.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2021-07-26T06:05:07.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:07.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2021-07-26T06:05:08.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:08.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2021-07-26T06:05:08.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:08.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2021-07-26T06:05:08.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:08.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2021-07-26T06:05:08.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:08.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2021-07-26T06:05:08.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:08.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2021-07-26T06:05:09.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:09.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2021-07-26T06:05:09.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:09.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2021-07-26T06:05:09.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:09.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2021-07-26T06:05:09.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:09.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2021-07-26T06:05:09.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:09.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2021-07-26T06:05:09.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:09.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2021-07-26T06:05:09.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:09.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2021-07-26T06:05:10.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:10.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2021-07-26T06:05:10.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:10.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2021-07-26T06:05:10.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:10.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2021-07-26T06:05:10.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:10.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2021-07-26T06:05:10.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:10.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2021-07-26T06:05:10.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:10.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2021-07-26T06:05:11.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:11.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2021-07-26T06:05:11.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:11.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2021-07-26T06:05:11.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:11.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2021-07-26T06:05:11.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:11.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2021-07-26T06:05:11.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:11.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2021-07-26T06:05:11.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:11.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2021-07-26T06:05:11.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:11.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2021-07-26T06:05:12.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:12.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2021-07-26T06:05:12.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:12.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2021-07-26T06:05:12.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:12.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2021-07-26T06:05:12.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:12.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2021-07-26T06:05:12.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:12.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2021-07-26T06:05:12.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:12.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2021-07-26T06:05:12.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:12.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2021-07-26T06:05:13.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:13.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2021-07-26T06:05:13.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:13.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2021-07-26T06:05:13.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:13.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2021-07-26T06:05:13.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:13.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2021-07-26T06:05:13.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:13.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2021-07-26T06:05:13.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:13.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2021-07-26T06:05:13.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:13.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2021-07-26T06:05:14.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:14.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2021-07-26T06:05:14.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:14.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2021-07-26T06:05:14.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:14.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2021-07-26T06:05:14.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:14.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2021-07-26T06:05:14.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:14.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2021-07-26T06:05:14.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:14.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2021-07-26T06:05:14.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:14.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2021-07-26T06:05:15.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:15.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2021-07-26T06:05:15.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:15.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2021-07-26T06:05:15.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:15.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2021-07-26T06:05:15.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:15.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2021-07-26T06:05:15.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:15.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2021-07-26T06:05:15.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:15.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2021-07-26T06:05:16.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:16.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2021-07-26T06:05:16.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:16.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2021-07-26T06:05:16.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-07-26T06:05:16.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-07-26T06:05:16.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:05:16.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2021-07-26T06:05:16.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:05:16.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:05:16.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:05:16.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:05:16.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:05:16.548 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:05:16.548 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:05:16.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:05:16.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:05:16.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:05:16.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:05:16.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:05:16.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:05:16.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:05:16.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:05:17.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:05:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:05:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:05:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:05:17.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:05:17.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:05:17.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836498 2021-07-26T06:05:17.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836498 2021-07-26T06:05:17.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498' 2021-07-26T06:05:17.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:05:17.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:05:17.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672975 2021-07-26T06:05:17.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672975 2021-07-26T06:05:17.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672975' 2021-07-26T06:05:17.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:05:17.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:05:17.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509453 2021-07-26T06:05:17.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509453 2021-07-26T06:05:17.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453' 2021-07-26T06:05:17.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:05:17.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836498 2021-07-26T06:05:17.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:05:17.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:05:17.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836498 2021-07-26T06:05:17.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:05:17.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836498 2021-07-26T06:05:17.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2021-07-26T06:05:17.545 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836498 2021-07-26T06:05:17.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:05:17.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836498 2021-07-26T06:05:17.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:05:18.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:05:18.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:05:19.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836498 -lt 21474836498 2021-07-26T06:05:19.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:05:19.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672975 2021-07-26T06:05:19.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:05:19.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:05:19.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:05:19.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672975 2021-07-26T06:05:19.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672975 2021-07-26T06:05:19.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2021-07-26T06:05:19.295 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672975 2021-07-26T06:05:19.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:05:19.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672976 -lt 42949672975 2021-07-26T06:05:19.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:05:19.745 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509453 2021-07-26T06:05:19.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:05:19.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:05:19.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509453 2021-07-26T06:05:19.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:05:19.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509453 2021-07-26T06:05:19.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-07-26T06:05:19.752 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509453 2021-07-26T06:05:19.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:05:20.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509453 2021-07-26T06:05:20.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:05:20.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:20.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:20.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:05:20.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:05:20.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:05:20.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:05:20.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:05:20.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:05:20.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:05:20.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:05:21.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:05:21.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:05:21.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:21.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:21.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:05:21.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:05:21.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:05:21.538 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2021-07-26T06:05:21.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T06:05:21.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=existing_1 2021-07-26T06:05:21.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test existing_1 2021-07-26T06:05:21.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:05:21.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2021-07-26T06:05:21.940 INFO:tasks.workunit.client.0.smithi094.stdout:blackholing osds 0 2 2021-07-26T06:05:21.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2021-07-26T06:05:21.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2021-07-26T06:05:22.326 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:05:22.327 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2021-07-26T06:04:51.154745+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:05:22.327 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:05:22.327 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:05:22.329 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:05:22.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:22.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2021-07-26T06:05:22.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:05:22.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:05:22.340 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:05:22.340 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:05:22.341 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:05:22.341 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:05:22.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-07-26T06:05:22.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok config set objectstore_blackhole 1 2021-07-26T06:05:22.424 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:05:22.425 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:05:22.425 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:05:22.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:22.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2021-07-26T06:05:22.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:05:22.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:05:22.434 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:05:22.435 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:05:22.435 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:05:22.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:05:22.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-07-26T06:05:22.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok config set objectstore_blackhole 1 2021-07-26T06:05:22.514 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:05:22.515 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:05:22.515 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:05:22.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2021-07-26T06:05:22.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2021-07-26T06:05:22.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2021-07-26T06:05:22.525 INFO:tasks.workunit.client.0.smithi094.stdout:writing divergent object 2021-07-26T06:05:22.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2021-07-26T06:05:22.526 INFO:tasks.workunit.client.0.smithi094.stdout:create missing divergent object 2021-07-26T06:05:22.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2021-07-26T06:05:22.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-07-26T06:05:22.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2021-07-26T06:05:22.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local pooltype=rep 2021-07-26T06:05:22.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-07-26T06:05:22.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local which=data 2021-07-26T06:05:22.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-07-26T06:05:22.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local poolname=test 2021-07-26T06:05:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-07-26T06:05:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local objname=existing_99 2021-07-26T06:05:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-07-26T06:05:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local dir=td/divergent-priors 2021-07-26T06:05:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: shift 2021-07-26T06:05:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local shard_id=0 2021-07-26T06:05:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shift 2021-07-26T06:05:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-07-26T06:05:22.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: get_osds test existing_99 2021-07-26T06:05:22.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=test 2021-07-26T06:05:22.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=existing_99 2021-07-26T06:05:22.532 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map test existing_99 2021-07-26T06:05:22.532 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T06:05:22.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T06:05:22.873 INFO:tasks.workunit.client.0.smithi094.stderr:0 2021-07-26T06:05:22.874 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:05:22.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2 2021-07-26T06:05:22.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: local -a initial_osds 2021-07-26T06:05:22.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: local osd_id=1 2021-07-26T06:05:22.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: '[' rep '!=' ec ']' 2021-07-26T06:05:22.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: shard_id= 2021-07-26T06:05:22.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: cat td/divergent-priors/1/type 2021-07-26T06:05:22.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: type=bluestore 2021-07-26T06:05:22.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2254: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-07-26T06:05:22.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: set_config: local daemon=osd 2021-07-26T06:05:22.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: set_config: local id=1 2021-07-26T06:05:22.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: local config=bluestore_debug_inject_read_err 2021-07-26T06:05:22.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: local value=true 2021-07-26T06:05:22.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: jq 'has("success")' 2021-07-26T06:05:22.879 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: get_asok_path osd.1 2021-07-26T06:05:22.879 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:05:22.880 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:05:22.881 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:05:22.881 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:05:22.881 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:05:22.881 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:05:22.881 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.18330/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-07-26T06:05:22.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: set_config: test true == true 2021-07-26T06:05:22.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: inject_eio: local loop=0 2021-07-26T06:05:22.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: inject_eio: grep -q Invalid 2021-07-26T06:05:22.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: get_asok_path osd.1 2021-07-26T06:05:22.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:05:22.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:05:22.972 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:05:22.972 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:05:22.972 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:05:22.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:05:22.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: CEPH_ARGS= 2021-07-26T06:05:22.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: inject_eio: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.1.asok injectdataerr test existing_99 2021-07-26T06:05:23.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2021-07-26T06:05:23.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2021-07-26T06:05:33.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2021-07-26T06:05:33.090 INFO:tasks.workunit.client.0.smithi094.stderr:rados: no process found 2021-07-26T06:05:33.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2021-07-26T06:05:33.091 INFO:tasks.workunit.client.0.smithi094.stdout:killing all the osds 2021-07-26T06:05:33.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2021-07-26T06:05:33.442 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:05:33.443 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 active+clean 2021-07-26T06:05:23.453094+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:05:33.443 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:05:33.443 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:05:33.445 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:05:33.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2021-07-26T06:05:33.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:05:33.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:05:33.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:05:33.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:05:33.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:05:33.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:05:33.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-07-26T06:05:33.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2021-07-26T06:05:34.057 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:05:34.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-07-26T06:05:34.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2021-07-26T06:05:34.599 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T06:05:34.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-07-26T06:05:34.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2021-07-26T06:05:35.250 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already down. 2021-07-26T06:05:35.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:35.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2021-07-26T06:05:35.817 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T06:05:35.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:35.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2021-07-26T06:05:36.447 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already out. 2021-07-26T06:05:36.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2021-07-26T06:05:36.456 INFO:tasks.workunit.client.0.smithi094.stdout:bringing up non_divergent 0 2 2021-07-26T06:05:36.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2021-07-26T06:05:36.842 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:05:36.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 stale+active+clean 2021-07-26T06:05:23.453094+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:05:36.843 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:05:36.843 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:05:36.845 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:05:36.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:36.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2021-07-26T06:05:36.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:05:36.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:05:36.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:05:36.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:05:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:05:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:05:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:05:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:05:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:05:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:05:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:05:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:05:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:05:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:05:36.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:05:36.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:05:36.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:05:36.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:05:36.860 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:05:36.861 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:05:36.861 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:05:36.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:05:36.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:05:36.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:05:36.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:05:36.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:05:36.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:05:36.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:05:36.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:05:36.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:05:36.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:05:36.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:05:36.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:05:36.865 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:05:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:05:36.867 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/0/whoami 2021-07-26T06:05:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:05:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:05:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:05:36.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:05:36.893 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:36.901+0000 7f25296c0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:05:36.896 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:36.901+0000 7f25296c0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:05:36.897 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:36.905+0000 7f25296c0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:05:37.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:05:37.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:05:37.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:05:37.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:05:37.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:05:37.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:37.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:05:37.261 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:05:37.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:37.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:05:37.435 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:37.441+0000 7f25296c0f40 -1 Falling back to public interface 2021-07-26T06:05:37.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:05:38.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:05:38.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:38.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:05:38.611 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:05:38.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:38.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:05:38.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:05:39.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:05:39.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:39.981 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:05:39.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:05:39.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:39.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:05:40.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:05:40.575 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:40.581+0000 7f25296c0f40 -1 osd.0 29 log_to_monitors {default=true} 2021-07-26T06:05:41.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:05:41.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:41.489 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:05:41.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:05:41.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:41.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:05:41.860 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/26323,v1:127.0.0.1:6803/26323] [v2:127.0.0.1:6804/26323,v1:127.0.0.1:6805/26323] exists,up 827849e5-5a86-4978-bf98-6b5fbb0f3a6b 2021-07-26T06:05:41.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:05:41.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:05:41.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:05:41.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:41.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2021-07-26T06:05:41.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:05:41.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:05:41.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:05:41.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:05:41.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:05:41.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:05:41.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:05:41.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:05:41.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:05:41.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:05:41.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:05:41.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:05:41.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:05:41.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:05:41.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:05:41.867 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:05:41.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:05:41.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:05:41.868 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:05:41.868 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:05:41.869 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:05:41.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:05:41.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:05:41.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:05:41.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:05:41.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:05:41.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:05:41.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:05:41.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:05:41.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:05:41.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:05:41.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:05:41.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:05:41.873 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:05:41.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:05:41.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/2/whoami 2021-07-26T06:05:41.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:05:41.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:05:41.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:05:41.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:05:41.903 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:41.909+0000 7ff21243ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:05:41.958 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:41.965+0000 7ff21243ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:05:41.960 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:41.969+0000 7ff21243ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:05:42.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:05:42.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:05:42.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:05:42.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:05:42.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:05:42.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:42.267 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:05:42.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:05:42.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:42.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:05:42.511 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:42.517+0000 7ff21243ef40 -1 Falling back to public interface 2021-07-26T06:05:42.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:05:43.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:05:43.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:43.664 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:05:43.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:05:43.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:43.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:05:44.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:05:45.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:05:45.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:45.071 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:05:45.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:05:45.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:45.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:05:45.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:05:45.714 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:05:45.721+0000 7ff21243ef40 -1 osd.2 29 log_to_monitors {default=true} 2021-07-26T06:05:46.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:05:46.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:05:46.445 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:05:46.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:05:46.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:05:46.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:05:46.848 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/26903,v1:127.0.0.1:6811/26903] [v2:127.0.0.1:6812/26903,v1:127.0.0.1:6813/26903] exists,up 1017022a-e6dc-4403-80b8-d7ce70ebba56 2021-07-26T06:05:46.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:05:46.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:05:46.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:05:46.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:46.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2021-07-26T06:05:47.440 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already in. 2021-07-26T06:05:47.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-07-26T06:05:47.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2021-07-26T06:05:48.075 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already in. 2021-07-26T06:05:48.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:05:48.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2021-07-26T06:05:48.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:05:48.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:05:48.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:05:48.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:05:48.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:05:48.086 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:05:48.086 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:05:48.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:05:48.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:05:48.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:05:48.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:05:48.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:05:48.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:05:48.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:05:48.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:05:48.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:05:48.800 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:05:48.800 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:05:48.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:05:48.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:05:48.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:05:48.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=154618822660 2021-07-26T06:05:48.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 154618822660 2021-07-26T06:05:48.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-154618822660' 2021-07-26T06:05:48.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:05:48.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:05:49.084 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-07-26T06:05:49.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T06:05:49.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T06:05:49.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T06:05:49.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:05:49.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:05:49.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=163208757251 2021-07-26T06:05:49.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 163208757251 2021-07-26T06:05:49.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2021-07-26T06:05:49.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:05:49.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-154618822660 2021-07-26T06:05:49.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:05:49.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:05:49.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-154618822660 2021-07-26T06:05:49.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:05:49.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=154618822660 2021-07-26T06:05:49.250 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 154618822660 2021-07-26T06:05:49.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2021-07-26T06:05:49.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:05:49.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 154618822658 -lt 154618822660 2021-07-26T06:05:49.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:05:50.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:05:50.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:05:51.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 154618822660 -lt 154618822660 2021-07-26T06:05:51.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:05:51.058 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-163208757251 2021-07-26T06:05:51.058 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:05:51.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:05:51.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-163208757251 2021-07-26T06:05:51.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:05:51.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=163208757251 2021-07-26T06:05:51.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2021-07-26T06:05:51.064 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 163208757251 2021-07-26T06:05:51.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:05:51.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757251 -lt 163208757251 2021-07-26T06:05:51.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:05:51.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:51.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:51.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:05:51.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:05:51.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:05:51.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:05:51.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:05:51.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:05:51.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:05:51.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:05:52.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:05:52.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:05:52.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:52.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:52.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:05:52.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:05:52.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:05:52.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:05:52.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:05:52.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:05:52.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:05:52.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:05:52.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:05:52.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:05:52.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:05:52.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:05:52.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:05:52.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:05:53.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:05:53.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:05:53.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:53.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:53.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:05:53.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:05:53.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:05:53.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:05:53.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:05:53.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:05:53.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:05:53.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:05:53.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:05:54.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:05:54.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:05:54.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:05:54.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:05:54.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:05:54.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:05:54.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:05:54.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:05:54.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:05:54.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:05:54.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:05:54.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:05:54.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:05:54.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:05:54.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:05:54.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:54.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:55.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:05:55.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:05:55.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:05:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:05:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:05:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:05:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:05:55.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:05:55.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:05:55.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:05:55.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:05:55.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:05:55.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:05:55.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:05:56.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:05:56.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:05:56.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:05:56.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:05:56.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:05:56.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:05:56.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:05:56.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:05:56.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:05:56.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:05:56.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:56.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:57.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:05:57.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:05:57.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:05:57.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:05:57.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:05:57.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:05:57.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:05:57.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:05:57.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:05:57.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:05:57.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:05:57.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:05:57.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:05:57.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:05:58.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:05:58.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:05:58.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:05:58.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:05:58.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:05:58.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:05:58.286 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:05:58.286 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:05:58.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:05:58.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:05:58.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:05:58.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:05:59.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:05:59.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:05:59.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:05:59.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:05:59.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:05:59.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:05:59.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:05:59.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:05:59.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:05:59.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:05:59.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:05:59.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 4 >= 8 )) 2021-07-26T06:05:59.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:05:59.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 1.6 2021-07-26T06:06:01.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:06:01.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:06:01.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:06:01.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:06:01.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:06:01.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:06:01.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:06:01.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:06:01.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:06:01.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:06:01.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:06:01.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:06:02.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:06:02.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:06:02.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:06:02.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:06:02.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:06:02.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:06:02.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:06:02.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:06:02.013 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:06:02.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:06:02.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:06:02.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 5 >= 8 )) 2021-07-26T06:06:02.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:06:02.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 3.2 2021-07-26T06:06:05.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:06:05.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:06:05.710 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:06:05.711 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:06:05.711 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:06:05.711 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:06:05.711 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:06:05.712 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:06:06.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:06:06.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:06:06.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:06:06.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:06:06.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:06:06.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:06:06.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:06:06.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:06:06.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:06:06.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:06:06.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:06:06.646 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:06:06.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:06:07.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:06:07.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:06:07.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 6 >= 8 )) 2021-07-26T06:06:07.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:06:07.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 6.4 2021-07-26T06:06:13.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:06:13.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:06:13.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:06:13.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:06:13.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:06:13.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:06:13.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:06:13.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:06:13.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:06:13.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:06:13.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:06:13.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:06:14.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:06:14.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:06:14.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:06:14.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:06:14.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:06:14.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:06:14.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:06:14.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:06:14.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:06:14.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:06:14.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:06:14.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 7 >= 8 )) 2021-07-26T06:06:14.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:06:14.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 7.3 2021-07-26T06:06:22.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:06:22.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:06:22.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:06:22.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:06:22.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:06:22.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:06:22.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:06:22.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:06:22.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:06:22.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:06:22.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:06:22.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:06:23.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:06:23.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:06:23.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:06:23.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:06:23.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:06:23.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:06:23.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:06:23.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:06:23.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:06:23.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:06:23.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:06:23.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 8 >= 8 )) 2021-07-26T06:06:23.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph report 2021-07-26T06:06:24.152 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:06:24.152 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_fingerprint": "612eb32e-2a11-450f-9edf-ff712143fafd", 2021-07-26T06:06:24.152 INFO:tasks.workunit.client.0.smithi094.stdout: "version": "17.0.0-6358-g197e564f", 2021-07-26T06:06:24.153 INFO:tasks.workunit.client.0.smithi094.stdout: "commit": "197e564fdaee12596de1dc8f3253d1704ae66d5d", 2021-07-26T06:06:24.153 INFO:tasks.workunit.client.0.smithi094.stdout: "timestamp": "2021-07-26T06:06:24.161114+0000", 2021-07-26T06:06:24.153 INFO:tasks.workunit.client.0.smithi094.stdout: "tag": "", 2021-07-26T06:06:24.153 INFO:tasks.workunit.client.0.smithi094.stdout: "health": { 2021-07-26T06:06:24.153 INFO:tasks.workunit.client.0.smithi094.stdout: "status": "HEALTH_WARN", 2021-07-26T06:06:24.154 INFO:tasks.workunit.client.0.smithi094.stdout: "checks": { 2021-07-26T06:06:24.154 INFO:tasks.workunit.client.0.smithi094.stdout: "OSDMAP_FLAGS": { 2021-07-26T06:06:24.154 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:06:24.154 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:06:24.154 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "nodown,noout,noin flag(s) set", 2021-07-26T06:06:24.155 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 17 2021-07-26T06:06:24.155 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.155 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [], 2021-07-26T06:06:24.155 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:06:24.155 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.156 INFO:tasks.workunit.client.0.smithi094.stdout: "OSD_DOWN": { 2021-07-26T06:06:24.156 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:06:24.156 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:06:24.156 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 osds down", 2021-07-26T06:06:24.156 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:06:24.157 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.157 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:06:24.157 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.157 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "osd.1 (root=default,host=smithi094) is down" 2021-07-26T06:06:24.158 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.158 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.158 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:06:24.158 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.158 INFO:tasks.workunit.client.0.smithi094.stdout: "PG_DEGRADED": { 2021-07-26T06:06:24.159 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:06:24.159 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:06:24.159 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-07-26T06:06:24.159 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:06:24.159 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.160 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:06:24.160 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.160 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:06:24.160 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.161 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.161 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:06:24.161 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.161 INFO:tasks.workunit.client.0.smithi094.stdout: "POOL_APP_NOT_ENABLED": { 2021-07-26T06:06:24.161 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:06:24.162 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:06:24.162 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 pool(s) do not have an application enabled", 2021-07-26T06:06:24.162 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:06:24.162 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.162 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:06:24.163 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.163 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "application not enabled on pool 'test'" 2021-07-26T06:06:24.163 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.163 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.164 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-07-26T06:06:24.164 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.164 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.164 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:06:24.164 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.165 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.165 INFO:tasks.workunit.client.0.smithi094.stdout: "mutes": [] 2021-07-26T06:06:24.165 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.165 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_first_committed": 1, 2021-07-26T06:06:24.165 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_last_committed": 1, 2021-07-26T06:06:24.166 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap": { 2021-07-26T06:06:24.166 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:06:24.166 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:06:24.166 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:04:08.308324Z", 2021-07-26T06:06:24.166 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:04:08.308324Z", 2021-07-26T06:06:24.167 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release": 17, 2021-07-26T06:06:24.167 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release_name": "quincy", 2021-07-26T06:06:24.167 INFO:tasks.workunit.client.0.smithi094.stdout: "election_strategy": 1, 2021-07-26T06:06:24.167 INFO:tasks.workunit.client.0.smithi094.stdout: "disallowed_leaders: ": "", 2021-07-26T06:06:24.168 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": false, 2021-07-26T06:06:24.168 INFO:tasks.workunit.client.0.smithi094.stdout: "features": { 2021-07-26T06:06:24.168 INFO:tasks.workunit.client.0.smithi094.stdout: "persistent": [ 2021-07-26T06:06:24.168 INFO:tasks.workunit.client.0.smithi094.stdout: "kraken", 2021-07-26T06:06:24.168 INFO:tasks.workunit.client.0.smithi094.stdout: "luminous", 2021-07-26T06:06:24.169 INFO:tasks.workunit.client.0.smithi094.stdout: "mimic", 2021-07-26T06:06:24.169 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap-prune", 2021-07-26T06:06:24.169 INFO:tasks.workunit.client.0.smithi094.stdout: "nautilus", 2021-07-26T06:06:24.169 INFO:tasks.workunit.client.0.smithi094.stdout: "octopus", 2021-07-26T06:06:24.169 INFO:tasks.workunit.client.0.smithi094.stdout: "pacific", 2021-07-26T06:06:24.170 INFO:tasks.workunit.client.0.smithi094.stdout: "elector-pinging", 2021-07-26T06:06:24.170 INFO:tasks.workunit.client.0.smithi094.stdout: "quincy" 2021-07-26T06:06:24.170 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.170 INFO:tasks.workunit.client.0.smithi094.stdout: "optional": [] 2021-07-26T06:06:24.170 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.171 INFO:tasks.workunit.client.0.smithi094.stdout: "mons": [ 2021-07-26T06:06:24.171 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.171 INFO:tasks.workunit.client.0.smithi094.stdout: "rank": 0, 2021-07-26T06:06:24.171 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "a", 2021-07-26T06:06:24.172 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:06:24.172 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.172 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.172 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.172 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115", 2021-07-26T06:06:24.173 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 0 2021-07-26T06:06:24.173 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.173 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.173 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.173 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115/0", 2021-07-26T06:06:24.174 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:7115/0", 2021-07-26T06:06:24.174 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 0, 2021-07-26T06:06:24.174 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 0, 2021-07-26T06:06:24.174 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_location": "{}" 2021-07-26T06:06:24.174 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.175 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.175 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.175 INFO:tasks.workunit.client.0.smithi094.stdout: "quorum": [ 2021-07-26T06:06:24.175 INFO:tasks.workunit.client.0.smithi094.stdout: 0 2021-07-26T06:06:24.176 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.176 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap": { 2021-07-26T06:06:24.176 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 42, 2021-07-26T06:06:24.176 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:06:24.176 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:04:09.215205+0000", 2021-07-26T06:06:24.177 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:05:48.283578+0000", 2021-07-26T06:06:24.177 INFO:tasks.workunit.client.0.smithi094.stdout: "last_up_change": "2021-07-26T06:05:46.146124+0000", 2021-07-26T06:06:24.177 INFO:tasks.workunit.client.0.smithi094.stdout: "last_in_change": "2021-07-26T06:05:47.919457+0000", 2021-07-26T06:06:24.177 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-07-26T06:06:24.177 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_num": 5800384, 2021-07-26T06:06:24.178 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_set": [ 2021-07-26T06:06:24.178 INFO:tasks.workunit.client.0.smithi094.stdout: "nodown", 2021-07-26T06:06:24.178 INFO:tasks.workunit.client.0.smithi094.stdout: "noin", 2021-07-26T06:06:24.178 INFO:tasks.workunit.client.0.smithi094.stdout: "noout", 2021-07-26T06:06:24.178 INFO:tasks.workunit.client.0.smithi094.stdout: "pglog_hardlimit", 2021-07-26T06:06:24.179 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snapdirs", 2021-07-26T06:06:24.179 INFO:tasks.workunit.client.0.smithi094.stdout: "recovery_deletes", 2021-07-26T06:06:24.179 INFO:tasks.workunit.client.0.smithi094.stdout: "sortbitwise" 2021-07-26T06:06:24.179 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.180 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_version": 9, 2021-07-26T06:06:24.180 INFO:tasks.workunit.client.0.smithi094.stdout: "full_ratio": 0.99000000953674316, 2021-07-26T06:06:24.180 INFO:tasks.workunit.client.0.smithi094.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-07-26T06:06:24.180 INFO:tasks.workunit.client.0.smithi094.stdout: "nearfull_ratio": 0.85000002384185791, 2021-07-26T06:06:24.180 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_snapshot": "", 2021-07-26T06:06:24.180 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_max": 1, 2021-07-26T06:06:24.181 INFO:tasks.workunit.client.0.smithi094.stdout: "max_osd": 3, 2021-07-26T06:06:24.181 INFO:tasks.workunit.client.0.smithi094.stdout: "require_min_compat_client": "luminous", 2021-07-26T06:06:24.181 INFO:tasks.workunit.client.0.smithi094.stdout: "min_compat_client": "jewel", 2021-07-26T06:06:24.181 INFO:tasks.workunit.client.0.smithi094.stdout: "require_osd_release": "quincy", 2021-07-26T06:06:24.181 INFO:tasks.workunit.client.0.smithi094.stdout: "pools": [ 2021-07-26T06:06:24.182 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.182 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:06:24.182 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_name": "test", 2021-07-26T06:06:24.182 INFO:tasks.workunit.client.0.smithi094.stdout: "create_time": "2021-07-26T06:04:49.066690+0000", 2021-07-26T06:06:24.183 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": 1, 2021-07-26T06:06:24.183 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_names": "hashpspool", 2021-07-26T06:06:24.183 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:06:24.183 INFO:tasks.workunit.client.0.smithi094.stdout: "size": 3, 2021-07-26T06:06:24.183 INFO:tasks.workunit.client.0.smithi094.stdout: "min_size": 2, 2021-07-26T06:06:24.184 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_rule": 0, 2021-07-26T06:06:24.184 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_count": 0, 2021-07-26T06:06:24.184 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_target": 0, 2021-07-26T06:06:24.184 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_barrier": 0, 2021-07-26T06:06:24.184 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-07-26T06:06:24.185 INFO:tasks.workunit.client.0.smithi094.stdout: "object_hash": 2, 2021-07-26T06:06:24.185 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_autoscale_mode": "off", 2021-07-26T06:06:24.185 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num": 1, 2021-07-26T06:06:24.185 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num": 1, 2021-07-26T06:06:24.185 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num_target": 1, 2021-07-26T06:06:24.186 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_target": 1, 2021-07-26T06:06:24.186 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_pending": 1, 2021-07-26T06:06:24.186 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pg_merge_meta": { 2021-07-26T06:06:24.186 INFO:tasks.workunit.client.0.smithi094.stdout: "source_pgid": "0.0", 2021-07-26T06:06:24.187 INFO:tasks.workunit.client.0.smithi094.stdout: "ready_epoch": 0, 2021-07-26T06:06:24.187 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_started": 0, 2021-07-26T06:06:24.187 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": 0, 2021-07-26T06:06:24.187 INFO:tasks.workunit.client.0.smithi094.stdout: "source_version": "0'0", 2021-07-26T06:06:24.188 INFO:tasks.workunit.client.0.smithi094.stdout: "target_version": "0'0" 2021-07-26T06:06:24.188 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.188 INFO:tasks.workunit.client.0.smithi094.stdout: "last_change": "29", 2021-07-26T06:06:24.188 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend": "0", 2021-07-26T06:06:24.188 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_prenautilus": "0", 2021-07-26T06:06:24.189 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_preluminous": "0", 2021-07-26T06:06:24.189 INFO:tasks.workunit.client.0.smithi094.stdout: "auid": 0, 2021-07-26T06:06:24.189 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_mode": "selfmanaged", 2021-07-26T06:06:24.189 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_seq": 0, 2021-07-26T06:06:24.190 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_epoch": 0, 2021-07-26T06:06:24.190 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_snaps": [], 2021-07-26T06:06:24.190 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps": "[]", 2021-07-26T06:06:24.190 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_bytes": 0, 2021-07-26T06:06:24.190 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_objects": 0, 2021-07-26T06:06:24.191 INFO:tasks.workunit.client.0.smithi094.stdout: "tiers": [], 2021-07-26T06:06:24.191 INFO:tasks.workunit.client.0.smithi094.stdout: "tier_of": -1, 2021-07-26T06:06:24.191 INFO:tasks.workunit.client.0.smithi094.stdout: "read_tier": -1, 2021-07-26T06:06:24.191 INFO:tasks.workunit.client.0.smithi094.stdout: "write_tier": -1, 2021-07-26T06:06:24.192 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_mode": "none", 2021-07-26T06:06:24.192 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_bytes": 0, 2021-07-26T06:06:24.192 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_objects": 0, 2021-07-26T06:06:24.192 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-07-26T06:06:24.192 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-07-26T06:06:24.193 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_full_ratio_micro": 800000, 2021-07-26T06:06:24.193 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_flush_age": 0, 2021-07-26T06:06:24.193 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_evict_age": 0, 2021-07-26T06:06:24.193 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profile": "", 2021-07-26T06:06:24.193 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_params": { 2021-07-26T06:06:24.194 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "none" 2021-07-26T06:06:24.194 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.194 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_period": 0, 2021-07-26T06:06:24.194 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_count": 0, 2021-07-26T06:06:24.195 INFO:tasks.workunit.client.0.smithi094.stdout: "use_gmt_hitset": true, 2021-07-26T06:06:24.195 INFO:tasks.workunit.client.0.smithi094.stdout: "min_read_recency_for_promote": 0, 2021-07-26T06:06:24.195 INFO:tasks.workunit.client.0.smithi094.stdout: "min_write_recency_for_promote": 0, 2021-07-26T06:06:24.195 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_grade_decay_rate": 0, 2021-07-26T06:06:24.195 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_search_last_n": 0, 2021-07-26T06:06:24.196 INFO:tasks.workunit.client.0.smithi094.stdout: "grade_table": [], 2021-07-26T06:06:24.196 INFO:tasks.workunit.client.0.smithi094.stdout: "stripe_width": 0, 2021-07-26T06:06:24.196 INFO:tasks.workunit.client.0.smithi094.stdout: "expected_num_objects": 0, 2021-07-26T06:06:24.196 INFO:tasks.workunit.client.0.smithi094.stdout: "fast_read": false, 2021-07-26T06:06:24.196 INFO:tasks.workunit.client.0.smithi094.stdout: "options": {}, 2021-07-26T06:06:24.197 INFO:tasks.workunit.client.0.smithi094.stdout: "application_metadata": {} 2021-07-26T06:06:24.197 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.197 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.197 INFO:tasks.workunit.client.0.smithi094.stdout: "osds": [ 2021-07-26T06:06:24.198 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.198 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:06:24.198 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "827849e5-5a86-4978-bf98-6b5fbb0f3a6b", 2021-07-26T06:06:24.198 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:06:24.198 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:06:24.199 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:06:24.199 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:06:24.199 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 5, 2021-07-26T06:06:24.199 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:06:24.199 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 36, 2021-07-26T06:06:24.200 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 41, 2021-07-26T06:06:24.200 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 30, 2021-07-26T06:06:24.200 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:06:24.200 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:06:24.201 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.201 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.201 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.201 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6802", 2021-07-26T06:06:24.201 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.202 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.202 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.202 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.202 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6803", 2021-07-26T06:06:24.202 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.203 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.203 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.203 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.203 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:06:24.204 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.204 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.204 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.204 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6804", 2021-07-26T06:06:24.204 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.205 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.205 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.205 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.205 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6805", 2021-07-26T06:06:24.205 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.206 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.206 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.206 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.206 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:06:24.207 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.207 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.207 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.207 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6808", 2021-07-26T06:06:24.207 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.208 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.208 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.208 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.208 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6809", 2021-07-26T06:06:24.208 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.209 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.209 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.209 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.209 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:06:24.209 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.210 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.210 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.210 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6806", 2021-07-26T06:06:24.210 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.210 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.211 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.211 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.211 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6807", 2021-07-26T06:06:24.211 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26323 2021-07-26T06:06:24.212 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.212 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.212 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.212 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6803/26323", 2021-07-26T06:06:24.212 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6805/26323", 2021-07-26T06:06:24.213 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6809/26323", 2021-07-26T06:06:24.213 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6807/26323", 2021-07-26T06:06:24.213 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:06:24.213 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:06:24.213 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:06:24.214 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.214 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.214 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.214 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:06:24.214 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1", 2021-07-26T06:06:24.215 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 0, 2021-07-26T06:06:24.215 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:06:24.215 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:06:24.215 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:06:24.216 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 0, 2021-07-26T06:06:24.216 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 0, 2021-07-26T06:06:24.216 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 10, 2021-07-26T06:06:24.216 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 23, 2021-07-26T06:06:24.216 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 31, 2021-07-26T06:06:24.217 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:06:24.217 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:06:24.217 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.217 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.217 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.218 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:06:24.218 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.218 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.218 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.219 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.219 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:06:24.219 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.219 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.219 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.220 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.220 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:06:24.220 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.220 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.220 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.221 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:06:24.221 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.221 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.221 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.222 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.222 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:06:24.222 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.222 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.222 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.223 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.223 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:06:24.223 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.223 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.224 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.224 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:06:24.224 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.224 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.224 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.225 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.225 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:06:24.225 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.225 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.225 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.226 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.226 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:06:24.226 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.226 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.227 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.227 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:06:24.227 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.227 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.227 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.228 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.228 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:06:24.228 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 20846 2021-07-26T06:06:24.228 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.228 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.229 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.229 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/20846", 2021-07-26T06:06:24.229 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/20846", 2021-07-26T06:06:24.229 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/20846", 2021-07-26T06:06:24.230 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/20846", 2021-07-26T06:06:24.230 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:06:24.230 INFO:tasks.workunit.client.0.smithi094.stdout: "exists" 2021-07-26T06:06:24.230 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.230 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.231 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.231 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:06:24.231 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "1017022a-e6dc-4403-80b8-d7ce70ebba56", 2021-07-26T06:06:24.231 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:06:24.231 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:06:24.232 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:06:24.232 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:06:24.232 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 15, 2021-07-26T06:06:24.232 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:06:24.233 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 38, 2021-07-26T06:06:24.233 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 0, 2021-07-26T06:06:24.233 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 32, 2021-07-26T06:06:24.233 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:06:24.233 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:06:24.234 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.234 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.234 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.234 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:06:24.234 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.235 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.235 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.235 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.235 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:06:24.236 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.236 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.236 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.236 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.236 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:06:24.237 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.237 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.237 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.237 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:06:24.237 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.238 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.238 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.238 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.238 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:06:24.239 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.239 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.239 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.239 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.239 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:06:24.240 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.240 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.240 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.240 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:06:24.240 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.241 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.241 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.241 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.241 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:06:24.241 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.242 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.242 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.242 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.242 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:06:24.242 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:06:24.243 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.243 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:06:24.243 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:06:24.243 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.243 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.244 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.244 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:06:24.244 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:06:24.244 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 26903 2021-07-26T06:06:24.245 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.245 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.245 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.245 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/26903", 2021-07-26T06:06:24.245 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/26903", 2021-07-26T06:06:24.246 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/26903", 2021-07-26T06:06:24.246 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/26903", 2021-07-26T06:06:24.246 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:06:24.246 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:06:24.246 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:06:24.247 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.247 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.247 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.247 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_xinfo": [ 2021-07-26T06:06:24.247 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.248 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:06:24.248 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:05:33.899647+0000", 2021-07-26T06:06:24.248 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:06:24.248 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:06:24.248 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:06:24.249 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:06:24.249 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:04:20.713518+0000", 2021-07-26T06:06:24.249 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:06:24.249 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.250 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.250 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:06:24.250 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:05:34.415654+0000", 2021-07-26T06:06:24.250 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:06:24.250 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:06:24.251 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:06:24.251 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:06:24.251 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:04:32.553709+0000", 2021-07-26T06:06:24.251 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:06:24.251 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.252 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.252 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:06:24.252 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:05:34.972725+0000", 2021-07-26T06:06:24.252 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:06:24.252 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:06:24.253 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:06:24.253 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:06:24.253 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:04:44.806872+0000", 2021-07-26T06:06:24.253 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:06:24.254 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.254 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.254 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap": [], 2021-07-26T06:06:24.254 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap_items": [], 2021-07-26T06:06:24.254 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_temp": [], 2021-07-26T06:06:24.255 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_temp": [], 2021-07-26T06:06:24.255 INFO:tasks.workunit.client.0.smithi094.stdout: "blocklist": {}, 2021-07-26T06:06:24.255 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profiles": { 2021-07-26T06:06:24.255 INFO:tasks.workunit.client.0.smithi094.stdout: "default": { 2021-07-26T06:06:24.255 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:06:24.256 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "2", 2021-07-26T06:06:24.256 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:06:24.256 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van" 2021-07-26T06:06:24.256 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.257 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.257 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps_queue": [], 2021-07-26T06:06:24.257 INFO:tasks.workunit.client.0.smithi094.stdout: "new_removed_snaps": [], 2021-07-26T06:06:24.257 INFO:tasks.workunit.client.0.smithi094.stdout: "new_purged_snaps": [], 2021-07-26T06:06:24.257 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_node_flags": {}, 2021-07-26T06:06:24.258 INFO:tasks.workunit.client.0.smithi094.stdout: "device_class_flags": {}, 2021-07-26T06:06:24.258 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": { 2021-07-26T06:06:24.258 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_enabled": false, 2021-07-26T06:06:24.258 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_bucket_count": 0, 2021-07-26T06:06:24.258 INFO:tasks.workunit.client.0.smithi094.stdout: "degraded_stretch_mode": 0, 2021-07-26T06:06:24.259 INFO:tasks.workunit.client.0.smithi094.stdout: "recovering_stretch_mode": 0, 2021-07-26T06:06:24.259 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_bucket": 0 2021-07-26T06:06:24.259 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.259 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.259 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_metadata": [ 2021-07-26T06:06:24.260 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.260 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:06:24.260 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:06:24.260 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6804/26323,v1:127.0.0.1:6805/26323]", 2021-07-26T06:06:24.261 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:06:24.261 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:06:24.261 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:06:24.261 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:06:24.261 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:06:24.262 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:06:24.262 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:06:24.262 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:06:24.262 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:06:24.262 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-07-26T06:06:24.263 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:06:24.263 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:06:24.263 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:06:24.263 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:06:24.263 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:06:24.264 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:06:24.264 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:06:24.264 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:06:24.264 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:06:24.265 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:06:24.265 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:06:24.265 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:06:24.265 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:06:24.265 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:06:24.266 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:06:24.266 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6802/26323,v1:127.0.0.1:6803/26323]", 2021-07-26T06:06:24.266 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:06:24.266 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/26323,v1:127.0.0.1:6809/26323]", 2021-07-26T06:06:24.266 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/26323,v1:127.0.0.1:6807/26323]", 2021-07-26T06:06:24.267 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:06:24.267 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:06:24.267 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:06:24.267 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:06:24.268 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:06:24.268 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:06:24.268 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:06:24.268 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:06:24.268 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:06:24.269 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/0", 2021-07-26T06:06:24.269 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:06:24.269 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:06:24.269 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:06:24.269 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.270 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.270 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:06:24.270 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:06:24.270 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/20846,v1:127.0.0.1:6813/20846]", 2021-07-26T06:06:24.270 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:06:24.271 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:06:24.271 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:06:24.271 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:06:24.271 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:06:24.272 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:06:24.272 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:06:24.272 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:06:24.272 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:06:24.272 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-07-26T06:06:24.273 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:06:24.273 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:06:24.273 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:06:24.273 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:06:24.273 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:06:24.274 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:06:24.274 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:06:24.274 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:06:24.274 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:06:24.274 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:06:24.275 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:06:24.275 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:06:24.275 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:06:24.275 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:06:24.275 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:06:24.276 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/20846,v1:127.0.0.1:6811/20846]", 2021-07-26T06:06:24.276 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:06:24.276 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/20846,v1:127.0.0.1:6817/20846]", 2021-07-26T06:06:24.276 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/20846,v1:127.0.0.1:6815/20846]", 2021-07-26T06:06:24.277 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:06:24.277 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:06:24.277 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:06:24.277 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:06:24.277 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:06:24.278 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:06:24.278 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:06:24.278 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:06:24.278 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:06:24.278 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/1", 2021-07-26T06:06:24.279 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:06:24.279 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:06:24.279 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:06:24.279 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.280 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.280 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:06:24.280 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:06:24.280 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/26903,v1:127.0.0.1:6813/26903]", 2021-07-26T06:06:24.280 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:06:24.281 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:06:24.281 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:06:24.281 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:06:24.281 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:06:24.281 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:06:24.282 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:06:24.282 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:06:24.282 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:06:24.282 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-07-26T06:06:24.282 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:06:24.283 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:06:24.283 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:06:24.283 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:06:24.283 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:06:24.284 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:06:24.284 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:06:24.284 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:06:24.284 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:06:24.284 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:06:24.285 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:06:24.285 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:06:24.285 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:06:24.285 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:06:24.285 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:06:24.286 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/26903,v1:127.0.0.1:6811/26903]", 2021-07-26T06:06:24.286 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:06:24.286 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/26903,v1:127.0.0.1:6817/26903]", 2021-07-26T06:06:24.286 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/26903,v1:127.0.0.1:6815/26903]", 2021-07-26T06:06:24.286 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:06:24.287 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:06:24.287 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:06:24.287 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:06:24.287 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:06:24.288 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:06:24.288 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:06:24.288 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:06:24.288 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:06:24.288 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/2", 2021-07-26T06:06:24.289 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:06:24.289 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:06:24.289 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:06:24.290 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.290 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.290 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_clean_epochs": { 2021-07-26T06:06:24.290 INFO:tasks.workunit.client.0.smithi094.stdout: "min_last_epoch_clean": 0, 2021-07-26T06:06:24.290 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": { 2021-07-26T06:06:24.291 INFO:tasks.workunit.client.0.smithi094.stdout: "per_pool": [] 2021-07-26T06:06:24.291 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.291 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_epochs": [ 2021-07-26T06:06:24.291 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.292 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:06:24.292 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 36 2021-07-26T06:06:24.292 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.292 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.292 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:06:24.293 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 10 2021-07-26T06:06:24.293 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.293 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.293 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:06:24.293 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 38 2021-07-26T06:06:24.294 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.294 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.294 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.294 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_first_committed": 1, 2021-07-26T06:06:24.294 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_last_committed": 42, 2021-07-26T06:06:24.295 INFO:tasks.workunit.client.0.smithi094.stdout: "crushmap": { 2021-07-26T06:06:24.295 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": [ 2021-07-26T06:06:24.295 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.295 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:06:24.296 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.0", 2021-07-26T06:06:24.296 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:06:24.296 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.296 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.296 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:06:24.297 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.1", 2021-07-26T06:06:24.297 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:06:24.297 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.297 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.297 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:06:24.298 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.2", 2021-07-26T06:06:24.298 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:06:24.298 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.298 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.298 INFO:tasks.workunit.client.0.smithi094.stdout: "types": [ 2021-07-26T06:06:24.299 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.299 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 0, 2021-07-26T06:06:24.299 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd" 2021-07-26T06:06:24.299 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.299 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.305 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:06:24.305 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "host" 2021-07-26T06:06:24.306 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.306 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.306 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 2, 2021-07-26T06:06:24.306 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "chassis" 2021-07-26T06:06:24.306 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.307 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.307 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 3, 2021-07-26T06:06:24.307 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "rack" 2021-07-26T06:06:24.307 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.308 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.308 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 4, 2021-07-26T06:06:24.308 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "row" 2021-07-26T06:06:24.308 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.308 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.309 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 5, 2021-07-26T06:06:24.309 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pdu" 2021-07-26T06:06:24.309 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.309 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.309 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 6, 2021-07-26T06:06:24.310 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pod" 2021-07-26T06:06:24.310 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.310 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.310 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 7, 2021-07-26T06:06:24.311 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "room" 2021-07-26T06:06:24.311 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.311 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.311 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 8, 2021-07-26T06:06:24.311 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "datacenter" 2021-07-26T06:06:24.312 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.312 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.312 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 9, 2021-07-26T06:06:24.312 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "zone" 2021-07-26T06:06:24.312 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.313 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.313 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 10, 2021-07-26T06:06:24.313 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "region" 2021-07-26T06:06:24.313 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.313 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.314 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:06:24.314 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "root" 2021-07-26T06:06:24.314 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.314 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.315 INFO:tasks.workunit.client.0.smithi094.stdout: "buckets": [ 2021-07-26T06:06:24.315 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.315 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -1, 2021-07-26T06:06:24.315 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default", 2021-07-26T06:06:24.315 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:06:24.316 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:06:24.316 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:06:24.316 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:06:24.316 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:06:24.316 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:06:24.317 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.317 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:06:24.317 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:06:24.317 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:06:24.318 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.318 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.318 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.318 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.318 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -2, 2021-07-26T06:06:24.319 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default~hdd", 2021-07-26T06:06:24.319 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:06:24.319 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:06:24.319 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:06:24.320 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:06:24.320 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:06:24.320 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:06:24.320 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.320 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:06:24.321 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:06:24.321 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:06:24.321 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.321 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.321 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.322 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.322 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:06:24.322 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094", 2021-07-26T06:06:24.322 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:06:24.323 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:06:24.323 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:06:24.323 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:06:24.323 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:06:24.323 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:06:24.324 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.324 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:06:24.324 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:06:24.324 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:06:24.324 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.325 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.325 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:06:24.325 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:06:24.325 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:06:24.326 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.326 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.326 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:06:24.326 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:06:24.326 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:06:24.327 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.327 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.327 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.327 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.327 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:06:24.328 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094~hdd", 2021-07-26T06:06:24.328 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:06:24.328 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:06:24.328 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:06:24.329 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:06:24.329 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:06:24.329 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:06:24.329 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.329 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:06:24.330 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:06:24.330 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:06:24.330 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.330 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.330 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:06:24.331 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:06:24.331 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:06:24.331 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.331 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.332 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:06:24.332 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:06:24.332 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:06:24.332 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.332 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.333 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.333 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.333 INFO:tasks.workunit.client.0.smithi094.stdout: "rules": [ 2021-07-26T06:06:24.333 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.333 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 0, 2021-07-26T06:06:24.334 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "replicated_rule", 2021-07-26T06:06:24.334 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:06:24.334 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:06:24.334 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.335 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:06:24.335 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:06:24.335 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:06:24.335 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.335 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.336 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_firstn", 2021-07-26T06:06:24.336 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:06:24.336 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:06:24.336 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.337 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.337 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:06:24.337 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.337 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:06:24.337 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.338 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.338 INFO:tasks.workunit.client.0.smithi094.stdout: "tunables": { 2021-07-26T06:06:24.338 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_tries": 0, 2021-07-26T06:06:24.338 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_fallback_tries": 0, 2021-07-26T06:06:24.338 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_total_tries": 50, 2021-07-26T06:06:24.339 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_descend_once": 1, 2021-07-26T06:06:24.339 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_vary_r": 1, 2021-07-26T06:06:24.339 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_stable": 1, 2021-07-26T06:06:24.339 INFO:tasks.workunit.client.0.smithi094.stdout: "straw_calc_version": 1, 2021-07-26T06:06:24.339 INFO:tasks.workunit.client.0.smithi094.stdout: "allowed_bucket_algs": 54, 2021-07-26T06:06:24.340 INFO:tasks.workunit.client.0.smithi094.stdout: "profile": "jewel", 2021-07-26T06:06:24.340 INFO:tasks.workunit.client.0.smithi094.stdout: "optimal_tunables": 1, 2021-07-26T06:06:24.340 INFO:tasks.workunit.client.0.smithi094.stdout: "legacy_tunables": 0, 2021-07-26T06:06:24.340 INFO:tasks.workunit.client.0.smithi094.stdout: "minimum_required_version": "jewel", 2021-07-26T06:06:24.340 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables": 1, 2021-07-26T06:06:24.341 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables2": 1, 2021-07-26T06:06:24.341 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v2_rules": 0, 2021-07-26T06:06:24.341 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables3": 1, 2021-07-26T06:06:24.341 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v3_rules": 0, 2021-07-26T06:06:24.341 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v4_buckets": 1, 2021-07-26T06:06:24.342 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables5": 1, 2021-07-26T06:06:24.342 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v5_rules": 0 2021-07-26T06:06:24.342 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.342 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_args": {} 2021-07-26T06:06:24.343 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.343 INFO:tasks.workunit.client.0.smithi094.stdout: "fsmap": { 2021-07-26T06:06:24.343 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:06:24.343 INFO:tasks.workunit.client.0.smithi094.stdout: "default_fscid": -1, 2021-07-26T06:06:24.343 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": { 2021-07-26T06:06:24.344 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": {}, 2021-07-26T06:06:24.344 INFO:tasks.workunit.client.0.smithi094.stdout: "ro_compat": {}, 2021-07-26T06:06:24.344 INFO:tasks.workunit.client.0.smithi094.stdout: "incompat": { 2021-07-26T06:06:24.344 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_1": "base v0.20", 2021-07-26T06:06:24.344 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_2": "client writeable ranges", 2021-07-26T06:06:24.345 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_3": "default file layouts on dirs", 2021-07-26T06:06:24.345 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_4": "dir inode in separate object", 2021-07-26T06:06:24.345 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_5": "mds uses versioned encoding", 2021-07-26T06:06:24.345 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_6": "dirfrag is stored in omap", 2021-07-26T06:06:24.346 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_8": "no anchor table", 2021-07-26T06:06:24.346 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_9": "file layout v2", 2021-07-26T06:06:24.346 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_10": "snaprealm v2" 2021-07-26T06:06:24.346 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.346 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.347 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_flags": { 2021-07-26T06:06:24.347 INFO:tasks.workunit.client.0.smithi094.stdout: "enable_multiple": true, 2021-07-26T06:06:24.347 INFO:tasks.workunit.client.0.smithi094.stdout: "ever_enabled_multiple": true 2021-07-26T06:06:24.347 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.347 INFO:tasks.workunit.client.0.smithi094.stdout: "standbys": [], 2021-07-26T06:06:24.348 INFO:tasks.workunit.client.0.smithi094.stdout: "filesystems": [] 2021-07-26T06:06:24.348 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.348 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_first_committed": 1, 2021-07-26T06:06:24.348 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_last_committed": 1, 2021-07-26T06:06:24.348 INFO:tasks.workunit.client.0.smithi094.stdout: "auth": { 2021-07-26T06:06:24.349 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:06:24.349 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 5, 2021-07-26T06:06:24.349 INFO:tasks.workunit.client.0.smithi094.stdout: "num_secrets": 3 2021-07-26T06:06:24.349 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.349 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:06:24.350 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_active": 1, 2021-07-26T06:06:24.350 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_unknown": 0, 2021-07-26T06:06:24.350 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osd": 3, 2021-07-26T06:06:24.350 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_sum": { 2021-07-26T06:06:24.351 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:06:24.351 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61700, 2021-07-26T06:06:24.351 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 100, 2021-07-26T06:06:24.351 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:06:24.351 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 300, 2021-07-26T06:06:24.352 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:06:24.352 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:06:24.352 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 100, 2021-07-26T06:06:24.352 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:06:24.352 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:06:24.353 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 100, 2021-07-26T06:06:24.353 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:06:24.353 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:06:24.353 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:06:24.353 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 100, 2021-07-26T06:06:24.354 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 100, 2021-07-26T06:06:24.354 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:06:24.354 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:06:24.354 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:06:24.355 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:06:24.355 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:06:24.355 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:06:24.355 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:06:24.355 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:06:24.356 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:06:24.356 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:06:24.356 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:06:24.356 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:06:24.356 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:06:24.357 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:06:24.357 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:06:24.357 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:06:24.357 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:06:24.358 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:06:24.358 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:06:24.358 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:06:24.358 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:06:24.358 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:06:24.359 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:06:24.359 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:06:24.359 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:06:24.359 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.360 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:06:24.360 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:06:24.360 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:06:24.360 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:06:24.360 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 0, 2021-07-26T06:06:24.361 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 0, 2021-07-26T06:06:24.361 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:06:24.361 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:06:24.361 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:06:24.361 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:06:24.362 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:06:24.362 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.362 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 16, 2021-07-26T06:06:24.362 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 16, 2021-07-26T06:06:24.362 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 2, 2021-07-26T06:06:24.363 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 2, 2021-07-26T06:06:24.363 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 0 2021-07-26T06:06:24.363 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.363 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum": { 2021-07-26T06:06:24.364 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:06:24.364 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:06:24.364 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:06:24.364 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:06:24.364 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:06:24.365 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:06:24.365 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:06:24.365 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18476, 2021-07-26T06:06:24.365 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2128, 2021-07-26T06:06:24.365 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:06:24.366 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16317, 2021-07-26T06:06:24.366 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554324, 2021-07-26T06:06:24.366 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:06:24.366 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:06:24.366 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103627776, 2021-07-26T06:06:24.367 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:06:24.367 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2179072, 2021-07-26T06:06:24.367 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 464140, 2021-07-26T06:06:24.367 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:06:24.368 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:06:24.368 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:06:24.368 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:06:24.368 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16709520 2021-07-26T06:06:24.368 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.369 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:06:24.369 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:06:24.369 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:06:24.369 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 1, 2021-07-26T06:06:24.369 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:06:24.370 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:06:24.370 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:06:24.370 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.370 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:06:24.370 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:06:24.371 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:06:24.371 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:06:24.371 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:06:24.371 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.371 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:06:24.372 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:06:24.372 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.372 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum_by_class": { 2021-07-26T06:06:24.372 INFO:tasks.workunit.client.0.smithi094.stdout: "hdd": { 2021-07-26T06:06:24.372 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:06:24.373 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:06:24.373 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:06:24.373 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:06:24.373 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:06:24.373 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:06:24.374 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:06:24.374 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18476, 2021-07-26T06:06:24.374 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2128, 2021-07-26T06:06:24.374 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:06:24.374 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16317, 2021-07-26T06:06:24.375 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554324, 2021-07-26T06:06:24.375 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:06:24.375 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:06:24.375 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103627776, 2021-07-26T06:06:24.375 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:06:24.376 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2179072, 2021-07-26T06:06:24.376 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 464140, 2021-07-26T06:06:24.376 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:06:24.376 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:06:24.377 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:06:24.377 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:06:24.377 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16709520 2021-07-26T06:06:24.377 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.377 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:06:24.377 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:06:24.378 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:06:24.378 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 1, 2021-07-26T06:06:24.378 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:06:24.378 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:06:24.379 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:06:24.379 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.379 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:06:24.379 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:06:24.379 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:06:24.380 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:06:24.380 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:06:24.380 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.380 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:06:24.380 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:06:24.381 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.381 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.381 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_stats": [ 2021-07-26T06:06:24.381 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.381 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:06:24.382 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:06:24.382 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:06:24.382 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61700, 2021-07-26T06:06:24.382 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 100, 2021-07-26T06:06:24.382 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:06:24.383 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 300, 2021-07-26T06:06:24.383 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:06:24.383 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:06:24.383 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 100, 2021-07-26T06:06:24.384 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:06:24.384 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:06:24.384 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 100, 2021-07-26T06:06:24.384 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:06:24.384 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:06:24.385 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:06:24.385 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 100, 2021-07-26T06:06:24.385 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 100, 2021-07-26T06:06:24.385 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:06:24.385 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:06:24.386 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:06:24.386 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:06:24.386 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:06:24.386 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:06:24.387 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:06:24.387 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:06:24.387 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:06:24.387 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:06:24.387 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:06:24.388 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:06:24.388 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:06:24.388 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:06:24.388 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:06:24.388 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:06:24.389 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:06:24.389 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:06:24.389 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:06:24.389 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:06:24.389 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:06:24.390 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:06:24.390 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:06:24.390 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:06:24.390 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:06:24.390 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.391 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:06:24.391 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:06:24.391 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:06:24.391 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:06:24.392 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 1228800, 2021-07-26T06:06:24.392 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 185100, 2021-07-26T06:06:24.392 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:06:24.392 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:06:24.392 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:06:24.393 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:06:24.393 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:06:24.393 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.393 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 16, 2021-07-26T06:06:24.393 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 16, 2021-07-26T06:06:24.394 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 2, 2021-07-26T06:06:24.394 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 2, 2021-07-26T06:06:24.394 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 3 2021-07-26T06:06:24.394 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.394 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.395 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_stats": [ 2021-07-26T06:06:24.395 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.395 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:06:24.395 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 154618822667 2021-07-26T06:06:24.396 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.396 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.396 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:06:24.396 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 42949672979 2021-07-26T06:06:24.396 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.397 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.397 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:06:24.397 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 163208757258 2021-07-26T06:06:24.397 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.397 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.398 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_state": [ 2021-07-26T06:06:24.398 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.398 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized+degraded", 2021-07-26T06:06:24.398 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 1 2021-07-26T06:06:24.399 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.399 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.399 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_osd": [ 2021-07-26T06:06:24.399 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.399 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:06:24.400 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:06:24.400 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:06:24.400 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:06:24.400 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.400 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.401 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:06:24.401 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 1, 2021-07-26T06:06:24.401 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:06:24.401 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:06:24.402 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.402 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.402 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:06:24.402 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:06:24.402 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 0, 2021-07-26T06:06:24.403 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:06:24.403 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.403 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.403 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": [ 2021-07-26T06:06:24.404 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:06:24.404 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:06:24.404 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": {} 2021-07-26T06:06:24.404 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.404 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:06:24.405 INFO:tasks.workunit.client.0.smithi094.stdout: "servicemap": { 2021-07-26T06:06:24.405 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 4, 2021-07-26T06:06:24.405 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:05:34.035382+0000", 2021-07-26T06:06:24.405 INFO:tasks.workunit.client.0.smithi094.stdout: "services": { 2021-07-26T06:06:24.405 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": { 2021-07-26T06:06:24.406 INFO:tasks.workunit.client.0.smithi094.stdout: "daemons": { 2021-07-26T06:06:24.406 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": "", 2021-07-26T06:06:24.406 INFO:tasks.workunit.client.0.smithi094.stdout: "2": { 2021-07-26T06:06:24.406 INFO:tasks.workunit.client.0.smithi094.stdout: "start_epoch": 0, 2021-07-26T06:06:24.407 INFO:tasks.workunit.client.0.smithi094.stdout: "start_stamp": "0.000000", 2021-07-26T06:06:24.407 INFO:tasks.workunit.client.0.smithi094.stdout: "gid": 0, 2021-07-26T06:06:24.407 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "(unrecognized address family 0)/0", 2021-07-26T06:06:24.407 INFO:tasks.workunit.client.0.smithi094.stdout: "metadata": {}, 2021-07-26T06:06:24.407 INFO:tasks.workunit.client.0.smithi094.stdout: "task_status": {} 2021-07-26T06:06:24.408 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.408 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.408 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.408 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.408 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:06:24.409 INFO:tasks.workunit.client.0.smithi094.stdout: "paxos": { 2021-07-26T06:06:24.409 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:06:24.409 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 398, 2021-07-26T06:06:24.409 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pn": 0, 2021-07-26T06:06:24.409 INFO:tasks.workunit.client.0.smithi094.stdout: "accepted_pn": 0 2021-07-26T06:06:24.410 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:06:24.410 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:06:24.411 INFO:tasks.workunit.client.0.smithi094.stderr:report 3184593474 2021-07-26T06:06:24.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: return 1 2021-07-26T06:06:24.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2021-07-26T06:06:24.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2021-07-26T06:06:24.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2021-07-26T06:06:24.412 INFO:tasks.workunit.client.0.smithi094.stdout:writing non-divergent object existing_4 2021-07-26T06:06:24.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:24.524 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:24.524 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2021-07-26T06:05:48.667316+0000 29'100 42:122 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:24.525 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:24.525 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:24.527 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:24.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2021-07-26T06:06:24.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2021-07-26T06:06:24.726 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery 2021-07-26T06:06:24.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:25.051 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:25.051 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2021-07-26T06:05:48.667316+0000 29'100 42:122 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:25.052 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:25.052 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:25.054 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:25.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-07-26T06:06:25.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2021-07-26T06:06:25.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:06:25.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:06:25.068 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:06:25.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:25.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:06:25.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:06:25.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-07-26T06:06:25.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok set_recovery_delay 100000 2021-07-26T06:06:25.154 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:06:25.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-07-26T06:06:25.163 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2021-07-26T06:06:25.163 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:06:25.163 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:06:25.164 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:06:25.164 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:25.164 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:06:25.165 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:06:25.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-07-26T06:06:25.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok set_recovery_delay 100000 2021-07-26T06:06:25.244 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:06:25.252 INFO:tasks.workunit.client.0.smithi094.stdout:revive divergent 1 2021-07-26T06:06:25.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2021-07-26T06:06:25.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:25.576 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:25.576 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2021-07-26T06:05:48.667316+0000 29'100 42:122 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:25.576 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:25.577 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:25.578 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:25.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2021-07-26T06:06:26.368 INFO:tasks.workunit.client.0.smithi094.stderr:noup is set 2021-07-26T06:06:26.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2021-07-26T06:06:26.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:06:26.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:06:26.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:06:26.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:06:26.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:06:26.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:06:26.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:06:26.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:06:26.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:06:26.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:06:26.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:06:26.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:06:26.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:06:26.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:06:26.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:06:26.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:06:26.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:06:26.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:06:26.387 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:06:26.387 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:26.387 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:06:26.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:06:26.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:06:26.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:06:26.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:06:26.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:06:26.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:06:26.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:06:26.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:06:26.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:06:26.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:06:26.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:06:26.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:06:26.392 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:06:26.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:06:26.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:06:26.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:06:26.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:06:26.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:06:26.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:06:26.419 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:26.425+0000 7f96cc467f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:26.434 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:26.441+0000 7f96cc467f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:26.436 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:26.445+0000 7f96cc467f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:26.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: activate_osd: return 0 2021-07-26T06:06:26.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2021-07-26T06:06:26.974 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:26.981+0000 7f96cc467f40 -1 Falling back to public interface 2021-07-26T06:06:29.770 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:29.777+0000 7f96cc467f40 -1 osd.1 29 log_to_monitors {default=true} 2021-07-26T06:06:31.808 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery divergent 2021-07-26T06:06:31.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2021-07-26T06:06:31.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:32.164 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:32.165 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:05:48.667316+0000 42'101 43:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:32.165 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:32.165 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:32.167 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:32.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2021-07-26T06:06:32.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:06:32.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:06:32.178 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:06:32.178 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:32.178 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:06:32.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:06:32.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2021-07-26T06:06:32.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.1.asok set_recovery_delay 100000 2021-07-26T06:06:32.259 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:06:32.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2021-07-26T06:06:33.080 INFO:tasks.workunit.client.0.smithi094.stderr:noup is unset 2021-07-26T06:06:33.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2021-07-26T06:06:33.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:06:33.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:06:33.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:06:33.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:06:33.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:06:33.094 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:06:33.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:06:33.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:06:33.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:06:33.677 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/26323,v1:127.0.0.1:6803/26323] [v2:127.0.0.1:6804/26323,v1:127.0.0.1:6805/26323] exists,up 827849e5-5a86-4978-bf98-6b5fbb0f3a6b 2021-07-26T06:06:33.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:06:33.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:06:33.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:06:33.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2021-07-26T06:06:33.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:06:33.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:06:33.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:06:33.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:06:33.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:06:33.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:06:33.681 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:06:33.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:06:33.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:06:34.087 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/28628,v1:127.0.0.1:6819/28628] [v2:127.0.0.1:6820/28628,v1:127.0.0.1:6821/28628] exists,up e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1 2021-07-26T06:06:34.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:06:34.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:06:34.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:06:34.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2021-07-26T06:06:34.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:06:34.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:06:34.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:06:34.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:06:34.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:06:34.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:06:34.091 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:06:34.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:06:34.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:06:34.471 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/26903,v1:127.0.0.1:6811/26903] [v2:127.0.0.1:6812/26903,v1:127.0.0.1:6813/26903] exists,up 1017022a-e6dc-4403-80b8-d7ce70ebba56 2021-07-26T06:06:34.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:06:34.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:06:34.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:06:34.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:34.842 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:34.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:05:48.667316+0000 42'101 43:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:34.842 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:34.843 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:34.845 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:34.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2021-07-26T06:06:34.853 INFO:tasks.workunit.client.0.smithi094.stdout:wait for peering 2021-07-26T06:06:34.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:35.196 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:35.196 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:05:48.667316+0000 42'101 43:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:35.196 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:35.197 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:35.198 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:35.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2021-07-26T06:06:44.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2021-07-26T06:06:44.806 INFO:tasks.workunit.client.0.smithi094.stdout:killing divergent 1 2021-07-26T06:06:44.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:45.156 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:45.157 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 2 2 0 0 61843 0 0 17 17 active+recovering+degraded+wait 2021-07-26T06:06:33.966519+0000 42'101 48:132 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:45.157 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:45.157 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:45.159 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:45.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2021-07-26T06:06:45.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:06:45.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:06:45.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:06:45.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:06:45.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:06:45.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:06:45.281 INFO:tasks.workunit.client.0.smithi094.stdout:reviving divergent 1 2021-07-26T06:06:45.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2021-07-26T06:06:45.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2021-07-26T06:06:45.629 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:06:45.630 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 2 2 0 0 61843 0 0 17 17 active+recovering+degraded+wait 2021-07-26T06:06:33.966519+0000 42'101 48:132 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:06:45.630 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:06:45.630 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:06:45.632 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:06:45.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2021-07-26T06:06:45.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:06:45.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:06:45.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:06:45.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:06:45.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:06:45.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:06:45.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:06:45.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:06:45.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:06:45.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:06:45.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:06:45.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:06:45.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:06:45.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:06:45.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:06:45.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:06:45.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:06:45.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:06:45.650 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:06:45.650 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:06:45.651 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:06:45.651 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:06:45.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:06:45.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:06:45.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:06:45.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:06:45.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:06:45.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:06:45.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:06:45.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:06:45.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:06:45.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:06:45.653 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:06:45.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:06:45.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:06:45.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:06:45.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:06:45.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:06:45.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:06:45.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:06:45.682 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:45.689+0000 7f96c4c59f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:45.685 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:45.693+0000 7f96c4c59f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:45.687 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:45.697+0000 7f96c4c59f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:06:46.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:06:46.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:06:46.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:06:46.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:06:46.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:06:46.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:06:46.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:06:46.008 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:06:46.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:06:46.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:06:46.222 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:46.229+0000 7f96c4c59f40 -1 Falling back to public interface 2021-07-26T06:06:46.394 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/28628,v1:127.0.0.1:6819/28628] [v2:127.0.0.1:6820/28628,v1:127.0.0.1:6821/28628] exists,up e86c5d24-f48a-44e1-aedb-f2ff2fcacfc1 2021-07-26T06:06:46.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:06:46.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:06:46.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:06:46.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2021-07-26T06:06:48.918 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:06:48.925+0000 7f96c4c59f40 -1 osd.1 48 log_to_monitors {default=true} 2021-07-26T06:07:06.397 INFO:tasks.workunit.client.0.smithi094.stdout:allowing recovery 2021-07-26T06:07:06.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2021-07-26T06:07:06.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2021-07-26T06:07:06.740 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:07:06.741 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 0 0 0 62460 0 0 18 18 active+clean 2021-07-26T06:06:50.989523+0000 48'102 52:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:07:06.741 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:07:06.741 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:07:06.743 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:07:06.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-07-26T06:07:06.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2021-07-26T06:07:06.889 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:07:06.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-07-26T06:07:06.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2021-07-26T06:07:07.041 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:07:07.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-07-26T06:07:07.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2021-07-26T06:07:07.217 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:07:07.225 INFO:tasks.workunit.client.0.smithi094.stdout:reading divergent objects 2021-07-26T06:07:07.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2021-07-26T06:07:07.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2021-07-26T06:07:07.566 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:07:07.566 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 0 0 0 62460 0 0 18 18 active+clean 2021-07-26T06:06:50.989523+0000 48'102 52:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:04:49.116950+0000 0'0 2021-07-26T06:04:49.116950+0000 0 2021-07-26T06:07:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:07:07.567 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:07:07.569 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:07:07.578 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2021-07-26T06:07:07.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2021-07-26T06:07:07.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:07:07.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2021-07-26T06:07:07.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:07:07.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2021-07-26T06:07:07.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:07:07.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2021-07-26T06:07:07.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:07:07.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2021-07-26T06:07:07.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2021-07-26T06:07:07.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-07-26T06:07:07.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:07:07.757 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4540.0:1 2021-07-26T06:05:22.565060+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2021-07-26T06:07:07.757 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4540.0:1 2021-07-26T06:05:22.565060+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2021-07-26T06:07:07.758 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:07:07.758 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2021-07-26T06:07:07.758 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-07-26T06:07:07.759 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:07:07.759 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2021-07-26T06:07:07.759 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4540.0:1 2021-07-26T06:05:22.565060+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2021-07-26T06:07:07.759 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:06:33.725+0000 7f96b2981700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-07-26T06:07:07.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-07-26T06:07:07.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:07:07.765 INFO:tasks.workunit.client.0.smithi094.stdout:success 2021-07-26T06:07:07.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2021-07-26T06:07:07.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2021-07-26T06:07:07.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T06:07:07.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T06:07:08.269 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T06:07:08.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2021-07-26T06:07:08.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:07:08.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:07:08.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:07:08.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:07:08.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:07:08.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:07:08.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-07-26T06:07:08.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:07:08.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:07:08.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:07:08.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:07:08.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:07:08.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:07:08.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:07:08.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:07:08.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:07:08.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:07:08.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:07:08.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:07:08.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:07:08.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:07:08.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:07:08.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:07:08.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:07:08.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:07:08.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:07:08.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:07:08.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:07:08.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:07:08.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:07:08.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:08.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:08.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:07:08.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:07:08.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:07:08.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-07-26T06:07:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-07-26T06:07:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-07-26T06:07:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-07-26T06:07:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:07:08.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:07:08.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:07:08.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:07:08.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:07:08.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:07:08.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:07:08.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:07:08.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:07:08.644 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:07:08.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:07:08.646 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:07:08.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:07:08.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:07:08.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:07:08.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:07:08.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:07:08.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:07:08.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:07:08.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:07:08.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:07:08.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:07:08.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:07:08.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:08.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:08.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:07:08.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:07:08.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:07:08.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-07-26T06:07:08.661 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:07:08.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:08.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:08.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.18330 2021-07-26T06:07:08.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:07:08.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:07:08.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:07:08.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-07-26T06:07:08.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2021-07-26T06:07:08.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2021-07-26T06:07:08.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2021-07-26T06:07:08.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2021-07-26T06:07:08.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2021-07-26T06:07:08.667 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2021-07-26T06:07:08.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2021-07-26T06:07:08.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2021-07-26T06:07:08.669 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:07:08.670 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:07:08.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2021-07-26T06:07:08.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-07-26T06:07:08.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:07:08.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:07:08.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:07:08.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-07-26T06:07:08.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-07-26T06:07:09.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:07:09.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:09.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:09.038 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:09.038 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:09.039 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:09.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:09.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:07:09.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:07:09.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:07:09.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:07:09.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:07:09.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:07:09.345 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:07:09.345 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:07:09.346 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:07:09.346 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:07:09.346 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:07:09.347 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:09.347 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:09.348 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:07:09.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:07:09.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get fsid 2021-07-26T06:07:09.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:07:09.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:07:09.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:07:09.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:07:09.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:07:09.437 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:07:09.437 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:07:09.437 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:07:09.438 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:07:09.438 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:09.438 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:09.438 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:07:09.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:07:09.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get mon_host 2021-07-26T06:07:09.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2021-07-26T06:07:09.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-07-26T06:07:09.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:07:09.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:07:09.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:07:09.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-07-26T06:07:09.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:07:10.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:07:10.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:10.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:10.417 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:10.417 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:10.417 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:10.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:10.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:07:10.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:07:10.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-07-26T06:07:10.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2021-07-26T06:07:10.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:07:10.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:07:10.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:07:10.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:07:10.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:07:10.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:07:10.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:07:10.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:07:10.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:07:10.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:07:10.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:07:10.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:07:10.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:07:10.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:07:10.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:07:10.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:07:10.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:10.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:10.460 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:10.460 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:10.460 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:10.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:10.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:10.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:07:10.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:07:10.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:07:10.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:07:10.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:07:10.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:07:10.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:07:10.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:10.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:07:10.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:07:10.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:07:10.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:07:10.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b02e7a44-f114-4850-90a6-03564e323691 2021-07-26T06:07:10.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b02e7a44-f114-4850-90a6-03564e323691' 2021-07-26T06:07:10.464 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 b02e7a44-f114-4850-90a6-03564e323691 2021-07-26T06:07:10.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:07:10.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOUf5gxRdyHRAAD74yZ1oC6g0RxCLCkSOlNg== 2021-07-26T06:07:10.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOUf5gxRdyHRAAD74yZ1oC6g0RxCLCkSOlNg=="}' 2021-07-26T06:07:10.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b02e7a44-f114-4850-90a6-03564e323691 -i td/divergent-priors/0/new.json 2021-07-26T06:07:10.878 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:07:10.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2021-07-26T06:07:10.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAOUf5gxRdyHRAAD74yZ1oC6g0RxCLCkSOlNg== --osd-uuid b02e7a44-f114-4850-90a6-03564e323691 2021-07-26T06:07:10.931 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:10.941+0000 7f763417ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:10.934 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:10.941+0000 7f763417ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:10.936 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:10.945+0000 7f763417ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:10.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:10.945+0000 7f763417ef40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:07:10.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:10.945+0000 7f763417ef40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:07:10.938 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:10.945+0000 7f763417ef40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:07:11.065 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:11.073+0000 7f763417ef40 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-07-26T06:07:16.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-07-26T06:07:16.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:07:16.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:07:16.593 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:07:16.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:07:17.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:07:17.062 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:07:17.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:07:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:07:17.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:07:17.100 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:17.109+0000 7f1413cdcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:17.113+0000 7f1413cdcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:17.106 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:17.113+0000 7f1413cdcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:17.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:07:17.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:07:17.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:07:17.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:07:17.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:07:17.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:17.438 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:07:17.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:07:17.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:17.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:17.653 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:17.661+0000 7f1413cdcf40 -1 Falling back to public interface 2021-07-26T06:07:17.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:18.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:18.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:18.810 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:07:18.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:07:18.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:18.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:19.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:20.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:20.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:20.196 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:07:20.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:07:20.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:20.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:20.220 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:20.229+0000 7f1413cdcf40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:07:20.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:21.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:21.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:21.617 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:07:21.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:07:21.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:21.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:07:22.040 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/31228,v1:127.0.0.1:6803/31228] [v2:127.0.0.1:6804/31228,v1:127.0.0.1:6805/31228] exists,up b02e7a44-f114-4850-90a6-03564e323691 2021-07-26T06:07:22.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:07:22.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:07:22.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:07:22.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-07-26T06:07:22.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2021-07-26T06:07:22.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:07:22.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:07:22.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:07:22.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:07:22.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:07:22.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:07:22.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:07:22.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:07:22.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:07:22.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:07:22.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:07:22.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:07:22.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:07:22.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:07:22.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:07:22.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:07:22.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:22.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:22.051 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:22.051 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:22.051 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:22.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:22.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:22.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:07:22.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:07:22.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:07:22.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:07:22.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:07:22.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:07:22.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:07:22.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:22.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:07:22.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:07:22.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:07:22.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:07:22.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a4d2525d-0097-4f8c-b13e-ee3a5e599d6d 2021-07-26T06:07:22.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a4d2525d-0097-4f8c-b13e-ee3a5e599d6d' 2021-07-26T06:07:22.056 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 a4d2525d-0097-4f8c-b13e-ee3a5e599d6d 2021-07-26T06:07:22.057 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:07:22.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaUf5gSQJGBRAAShPjR5RpSmY9XIghJ4FAQQ== 2021-07-26T06:07:22.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaUf5gSQJGBRAAShPjR5RpSmY9XIghJ4FAQQ=="}' 2021-07-26T06:07:22.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a4d2525d-0097-4f8c-b13e-ee3a5e599d6d -i td/divergent-priors/1/new.json 2021-07-26T06:07:22.616 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:07:22.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2021-07-26T06:07:22.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAaUf5gSQJGBRAAShPjR5RpSmY9XIghJ4FAQQ== --osd-uuid a4d2525d-0097-4f8c-b13e-ee3a5e599d6d 2021-07-26T06:07:22.666 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:22.677+0000 7fdbb144bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:22.672 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:22.681+0000 7fdbb144bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:22.674 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:22.681+0000 7fdbb144bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:22.675 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:22.685+0000 7fdbb144bf40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:07:22.676 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:22.685+0000 7fdbb144bf40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:07:22.676 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:22.685+0000 7fdbb144bf40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:07:22.717 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:22.725+0000 7fdbb144bf40 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-07-26T06:07:28.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-07-26T06:07:28.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:07:28.837 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:07:28.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:07:28.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:07:29.368 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:07:29.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:07:29.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:29.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:07:29.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:07:29.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:07:29.405 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:29.413+0000 7f3d1f811f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:29.407 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:29.417+0000 7f3d1f811f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:29.409 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:29.417+0000 7f3d1f811f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:29.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:07:29.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:07:29.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:07:29.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:07:29.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:07:29.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:29.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:07:29.799 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:07:29.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:07:29.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:29.957 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:29.965+0000 7f3d1f811f40 -1 Falling back to public interface 2021-07-26T06:07:30.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:31.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:31.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:31.274 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:07:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:07:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:07:31.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:32.264 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:32.273+0000 7f3d1f811f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:07:32.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:32.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:32.652 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:07:32.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:07:32.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:32.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:07:33.127 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/32635,v1:127.0.0.1:6811/32635] [v2:127.0.0.1:6812/32635,v1:127.0.0.1:6813/32635] exists,up a4d2525d-0097-4f8c-b13e-ee3a5e599d6d 2021-07-26T06:07:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:07:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:07:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:07:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-07-26T06:07:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2021-07-26T06:07:33.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:07:33.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:07:33.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:07:33.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:07:33.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:07:33.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:07:33.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:07:33.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:07:33.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:07:33.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:07:33.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:07:33.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:07:33.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:07:33.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:07:33.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:07:33.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:07:33.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:07:33.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:07:33.137 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:07:33.137 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:07:33.137 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:07:33.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:33.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:07:33.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:07:33.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:07:33.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:07:33.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:07:33.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:07:33.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:07:33.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:07:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:07:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:07:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:07:33.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:07:33.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b82adfb-7a16-42ca-ab0d-8102aa9df430 2021-07-26T06:07:33.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4b82adfb-7a16-42ca-ab0d-8102aa9df430' 2021-07-26T06:07:33.142 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 4b82adfb-7a16-42ca-ab0d-8102aa9df430 2021-07-26T06:07:33.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:07:33.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlUf5gCnJcChAAQ4XXiNTpb9IolehbbDnpsg== 2021-07-26T06:07:33.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlUf5gCnJcChAAQ4XXiNTpb9IolehbbDnpsg=="}' 2021-07-26T06:07:33.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b82adfb-7a16-42ca-ab0d-8102aa9df430 -i td/divergent-priors/2/new.json 2021-07-26T06:07:33.762 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:07:33.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2021-07-26T06:07:33.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAlUf5gCnJcChAAQ4XXiNTpb9IolehbbDnpsg== --osd-uuid 4b82adfb-7a16-42ca-ab0d-8102aa9df430 2021-07-26T06:07:33.815 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:33.825+0000 7f3ce2ab3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:33.834 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:33.841+0000 7f3ce2ab3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:33.836 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:33.845+0000 7f3ce2ab3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:33.837 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:33.845+0000 7f3ce2ab3f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:07:33.837 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:33.845+0000 7f3ce2ab3f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:07:33.838 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:33.845+0000 7f3ce2ab3f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:07:33.942 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:33.953+0000 7f3ce2ab3f40 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-07-26T06:07:39.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-07-26T06:07:39.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:07:39.853 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:07:39.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:07:39.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:07:40.375 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:07:40.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:07:40.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:07:40.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:07:40.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:07:40.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:07:40.412 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:40.421+0000 7f6bf601af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:40.414 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:40.425+0000 7f6bf601af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:40.416 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:40.425+0000 7f6bf601af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:07:40.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:07:40.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:07:40.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:07:40.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:07:40.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:07:40.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:40.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:07:40.736 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:07:40.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:40.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:40.960 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:40.969+0000 7f6bf601af40 -1 Falling back to public interface 2021-07-26T06:07:41.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:42.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:42.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:42.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:07:42.131 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:07:42.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:42.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:42.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:43.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:43.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:43.523 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:07:43.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:07:43.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:43.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:43.525 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:07:43.533+0000 7f6bf601af40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:07:44.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:07:45.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:07:45.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:07:45.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:07:45.056 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:07:45.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:07:45.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:07:45.428 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/34027,v1:127.0.0.1:6819/34027] [v2:127.0.0.1:6820/34027,v1:127.0.0.1:6821/34027] exists,up 4b82adfb-7a16-42ca-ab0d-8102aa9df430 2021-07-26T06:07:45.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:07:45.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:07:45.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:07:45.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2021-07-26T06:07:46.216 INFO:tasks.workunit.client.0.smithi094.stderr:noout is set 2021-07-26T06:07:46.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2021-07-26T06:07:46.903 INFO:tasks.workunit.client.0.smithi094.stderr:noin is set 2021-07-26T06:07:46.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2021-07-26T06:07:47.689 INFO:tasks.workunit.client.0.smithi094.stderr:nodown is set 2021-07-26T06:07:47.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2021-07-26T06:07:47.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T06:07:48.301 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T06:07:48.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:07:49.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2021-07-26T06:07:49.959 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 3 2021-07-26T06:07:49.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2021-07-26T06:07:50.803 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 min_size to 2 2021-07-26T06:07:50.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2021-07-26T06:07:50.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:07:50.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:07:51.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:07:51.197 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:07:51.197 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:07:51.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:07:51.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:51.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:07:51.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T06:07:51.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T06:07:51.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T06:07:51.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:51.346 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:07:51.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:07:51.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:07:51.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-07-26T06:07:51.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:51.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:07:51.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509445 2021-07-26T06:07:51.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509445 2021-07-26T06:07:51.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509445' 2021-07-26T06:07:51.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:51.646 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T06:07:51.646 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:51.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:07:51.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T06:07:51.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:51.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T06:07:51.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T06:07:51.651 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836488 2021-07-26T06:07:51.651 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:07:52.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836488 2021-07-26T06:07:52.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:07:53.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:07:53.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:07:53.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836488 2021-07-26T06:07:53.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:53.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:07:53.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:53.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:07:53.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:07:53.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:53.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:07:53.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:07:53.401 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672966 2021-07-26T06:07:53.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:07:53.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672966 2021-07-26T06:07:53.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:53.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509445 2021-07-26T06:07:53.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:53.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:07:53.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509445 2021-07-26T06:07:53.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:53.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509445 2021-07-26T06:07:53.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-07-26T06:07:53.786 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509445 2021-07-26T06:07:53.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:07:54.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509445 2021-07-26T06:07:54.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2021-07-26T06:07:54.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:07:54.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:07:54.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:07:54.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:07:54.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:07:54.167 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:07:54.167 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:07:54.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:07:54.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:07:54.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:07:54.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:07:54.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:07:54.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:07:54.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:07:54.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:07:54.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:07:54.832 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:07:54.832 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:07:54.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:07:54.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:54.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:07:54.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836491 2021-07-26T06:07:54.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836491 2021-07-26T06:07:54.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491' 2021-07-26T06:07:54.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:54.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:07:55.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672969 2021-07-26T06:07:55.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672969 2021-07-26T06:07:55.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2021-07-26T06:07:55.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:07:55.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:07:55.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509448 2021-07-26T06:07:55.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509448 2021-07-26T06:07:55.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448' 2021-07-26T06:07:55.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:55.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836491 2021-07-26T06:07:55.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:55.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:07:55.256 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836491 2021-07-26T06:07:55.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:55.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836491 2021-07-26T06:07:55.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-07-26T06:07:55.259 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836491 2021-07-26T06:07:55.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:07:55.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836491 2021-07-26T06:07:55.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:07:56.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:07:56.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:07:57.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836491 2021-07-26T06:07:57.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:57.008 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672969 2021-07-26T06:07:57.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:57.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:07:57.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672969 2021-07-26T06:07:57.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:57.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672969 2021-07-26T06:07:57.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-07-26T06:07:57.013 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672969 2021-07-26T06:07:57.013 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:07:57.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672969 -lt 42949672969 2021-07-26T06:07:57.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:07:57.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509448 2021-07-26T06:07:57.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:07:57.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:07:57.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509448 2021-07-26T06:07:57.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:07:57.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509448 2021-07-26T06:07:57.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-07-26T06:07:57.385 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509448 2021-07-26T06:07:57.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:07:57.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509448 -lt 64424509448 2021-07-26T06:07:57.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:07:57.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:07:57.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:07:58.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:07:58.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:07:58.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:07:58.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:07:58.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:07:58.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:07:58.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:07:58.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:07:58.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:07:58.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:07:58.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:07:58.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:07:59.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:07:59.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:07:59.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:07:59.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2021-07-26T06:07:59.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2021-07-26T06:07:59.410 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:07:59.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2021-07-26T06:07:59.418 INFO:tasks.workunit.client.0.smithi094.stdout:primary and soon to be divergent is 1 2021-07-26T06:07:59.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2021-07-26T06:07:59.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:07:59.769 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:07:59.770 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:07:49.053276+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:07:59.770 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:07:59.770 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:07:59.771 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:07:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2021-07-26T06:07:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-07-26T06:07:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2021-07-26T06:07:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2021-07-26T06:07:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-07-26T06:07:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2021-07-26T06:07:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2021-07-26T06:07:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-07-26T06:07:59.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2021-07-26T06:07:59.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2021-07-26T06:07:59.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2021-07-26T06:07:59.783 INFO:tasks.workunit.client.0.smithi094.stdout:writing initial objects 2021-07-26T06:07:59.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2021-07-26T06:07:59.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:07:59.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2021-07-26T06:07:59.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:07:59.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2021-07-26T06:08:00.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:00.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2021-07-26T06:08:00.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:00.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2021-07-26T06:08:00.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:00.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2021-07-26T06:08:00.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:00.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2021-07-26T06:08:00.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:00.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2021-07-26T06:08:00.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:00.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2021-07-26T06:08:00.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:00.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2021-07-26T06:08:01.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:01.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2021-07-26T06:08:01.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:01.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2021-07-26T06:08:01.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:01.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2021-07-26T06:08:01.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:01.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2021-07-26T06:08:01.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:01.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2021-07-26T06:08:01.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:01.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2021-07-26T06:08:02.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:02.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2021-07-26T06:08:02.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:02.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2021-07-26T06:08:02.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:02.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2021-07-26T06:08:02.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:02.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2021-07-26T06:08:02.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:02.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2021-07-26T06:08:02.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:02.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2021-07-26T06:08:02.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:02.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2021-07-26T06:08:03.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:03.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2021-07-26T06:08:03.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:03.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2021-07-26T06:08:03.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:03.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2021-07-26T06:08:03.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:03.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2021-07-26T06:08:03.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:03.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2021-07-26T06:08:03.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:03.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2021-07-26T06:08:03.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:03.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2021-07-26T06:08:04.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:04.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2021-07-26T06:08:04.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:04.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2021-07-26T06:08:04.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:04.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2021-07-26T06:08:04.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:04.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2021-07-26T06:08:04.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:04.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2021-07-26T06:08:04.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:04.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2021-07-26T06:08:05.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:05.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2021-07-26T06:08:05.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:05.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2021-07-26T06:08:05.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:05.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2021-07-26T06:08:05.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:05.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2021-07-26T06:08:05.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:05.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2021-07-26T06:08:05.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:05.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2021-07-26T06:08:05.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:05.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2021-07-26T06:08:06.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:06.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2021-07-26T06:08:06.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:06.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2021-07-26T06:08:06.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:06.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2021-07-26T06:08:06.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:06.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2021-07-26T06:08:06.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:06.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2021-07-26T06:08:06.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:06.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2021-07-26T06:08:06.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:06.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2021-07-26T06:08:07.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:07.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2021-07-26T06:08:07.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:07.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2021-07-26T06:08:07.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:07.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2021-07-26T06:08:07.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:07.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2021-07-26T06:08:07.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:07.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2021-07-26T06:08:07.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:07.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2021-07-26T06:08:08.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:08.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2021-07-26T06:08:08.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:08.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2021-07-26T06:08:08.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:08.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2021-07-26T06:08:08.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:08.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2021-07-26T06:08:08.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:08.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2021-07-26T06:08:08.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:08.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2021-07-26T06:08:09.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2021-07-26T06:08:09.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2021-07-26T06:08:09.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2021-07-26T06:08:09.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2021-07-26T06:08:09.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2021-07-26T06:08:09.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2021-07-26T06:08:09.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2021-07-26T06:08:09.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:09.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2021-07-26T06:08:10.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:10.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2021-07-26T06:08:10.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:10.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2021-07-26T06:08:10.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:10.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2021-07-26T06:08:10.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:10.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2021-07-26T06:08:10.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:10.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2021-07-26T06:08:10.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:10.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2021-07-26T06:08:11.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:11.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2021-07-26T06:08:11.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:11.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2021-07-26T06:08:11.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:11.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2021-07-26T06:08:11.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:11.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2021-07-26T06:08:11.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:11.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2021-07-26T06:08:11.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:11.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2021-07-26T06:08:11.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:11.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2021-07-26T06:08:12.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:12.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2021-07-26T06:08:12.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:12.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2021-07-26T06:08:12.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:12.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2021-07-26T06:08:12.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:12.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2021-07-26T06:08:12.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:12.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2021-07-26T06:08:12.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:12.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2021-07-26T06:08:12.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:12.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2021-07-26T06:08:13.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:13.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2021-07-26T06:08:13.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:13.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2021-07-26T06:08:13.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:13.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2021-07-26T06:08:13.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:13.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2021-07-26T06:08:13.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:13.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2021-07-26T06:08:13.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:13.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2021-07-26T06:08:13.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:13.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2021-07-26T06:08:14.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:14.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2021-07-26T06:08:14.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:14.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2021-07-26T06:08:14.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:14.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2021-07-26T06:08:14.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-07-26T06:08:14.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2021-07-26T06:08:14.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:08:14.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2021-07-26T06:08:14.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:08:14.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:08:14.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:08:14.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:08:14.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:08:14.640 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:08:14.640 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:08:14.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:08:14.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:08:14.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:08:14.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:08:14.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:08:14.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:08:14.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:08:14.769 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:08:15.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:08:15.254 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:08:15.254 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:08:15.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:08:15.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:08:15.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:08:15.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836497 2021-07-26T06:08:15.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836497 2021-07-26T06:08:15.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836497' 2021-07-26T06:08:15.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:08:15.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:08:15.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672975 2021-07-26T06:08:15.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672975 2021-07-26T06:08:15.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836497 1-42949672975' 2021-07-26T06:08:15.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:08:15.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:08:15.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509454 2021-07-26T06:08:15.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509454 2021-07-26T06:08:15.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836497 1-42949672975 2-64424509454' 2021-07-26T06:08:15.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:08:15.770 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836497 2021-07-26T06:08:15.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:08:15.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:08:15.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836497 2021-07-26T06:08:15.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:08:15.775 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836497 2021-07-26T06:08:15.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836497 2021-07-26T06:08:15.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2021-07-26T06:08:15.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:08:16.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836497 2021-07-26T06:08:16.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:08:17.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:08:17.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:08:17.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836497 -lt 21474836497 2021-07-26T06:08:17.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:08:17.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672975 2021-07-26T06:08:17.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:08:17.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:08:17.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672975 2021-07-26T06:08:17.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:08:17.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672975 2021-07-26T06:08:17.518 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672975 2021-07-26T06:08:17.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2021-07-26T06:08:17.519 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:08:17.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672975 -lt 42949672975 2021-07-26T06:08:17.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:08:17.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509454 2021-07-26T06:08:17.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:08:17.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:08:17.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509454 2021-07-26T06:08:17.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:08:17.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509454 2021-07-26T06:08:17.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-07-26T06:08:17.938 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509454 2021-07-26T06:08:17.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:08:18.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509454 -lt 64424509454 2021-07-26T06:08:18.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:08:18.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:18.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:18.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:08:18.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:08:18.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:08:18.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:08:18.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:08:18.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:08:18.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:08:18.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:08:19.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:08:19.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:08:19.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:19.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:19.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:08:19.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:08:19.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:08:19.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2021-07-26T06:08:19.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T06:08:19.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=existing_1 2021-07-26T06:08:19.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test existing_1 2021-07-26T06:08:19.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:08:20.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2021-07-26T06:08:20.012 INFO:tasks.workunit.client.0.smithi094.stdout:blackholing osds 0 2 2021-07-26T06:08:20.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2021-07-26T06:08:20.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:08:20.379 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:08:20.380 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2021-07-26T06:07:49.053276+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:08:20.380 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:08:20.381 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:08:20.382 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:08:20.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:20.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2021-07-26T06:08:20.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:08:20.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:08:20.401 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:08:20.401 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:08:20.401 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:08:20.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:08:20.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-07-26T06:08:20.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok config set objectstore_blackhole 1 2021-07-26T06:08:20.473 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:08:20.473 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:08:20.473 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:08:20.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:20.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2021-07-26T06:08:20.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:08:20.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:08:20.481 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:08:20.482 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:08:20.482 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:08:20.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:08:20.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-07-26T06:08:20.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok config set objectstore_blackhole 1 2021-07-26T06:08:20.551 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:08:20.551 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:08:20.551 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:08:20.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2021-07-26T06:08:20.558 INFO:tasks.workunit.client.0.smithi094.stdout:create new divergent objects 2021-07-26T06:08:20.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2021-07-26T06:08:20.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:08:20.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:08:20.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2021-07-26T06:08:20.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2021-07-26T06:08:20.561 INFO:tasks.workunit.client.0.smithi094.stdout:writing divergent objects 2021-07-26T06:08:20.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2021-07-26T06:08:20.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2021-07-26T06:08:20.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-07-26T06:08:20.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-07-26T06:08:20.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2021-07-26T06:08:20.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2021-07-26T06:08:20.564 INFO:tasks.workunit.client.0.smithi094.stdout:remove divergent objects 2021-07-26T06:08:20.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2021-07-26T06:08:20.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2021-07-26T06:08:20.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-07-26T06:08:20.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2021-07-26T06:08:20.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2021-07-26T06:08:20.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-07-26T06:08:20.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2021-07-26T06:08:20.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2021-07-26T06:08:20.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2021-07-26T06:08:20.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2021-07-26T06:08:20.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2021-07-26T06:08:30.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2021-07-26T06:08:30.575 INFO:tasks.workunit.client.0.smithi094.stderr:rados: no process found 2021-07-26T06:08:30.576 INFO:tasks.workunit.client.0.smithi094.stdout:killing all the osds 2021-07-26T06:08:30.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2021-07-26T06:08:30.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:08:30.924 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:08:30.925 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 0 0 0 60752 0 0 16 16 active+clean 2021-07-26T06:07:49.053276+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:08:30.926 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:08:30.926 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:08:30.927 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:08:30.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2021-07-26T06:08:30.942 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:08:30.942 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:08:30.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:08:30.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:08:30.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:08:31.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:08:31.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-07-26T06:08:31.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2021-07-26T06:08:31.586 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:08:31.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-07-26T06:08:31.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2021-07-26T06:08:32.178 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T06:08:32.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-07-26T06:08:32.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2021-07-26T06:08:32.762 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already down. 2021-07-26T06:08:32.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:32.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2021-07-26T06:08:33.339 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T06:08:33.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:33.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2021-07-26T06:08:33.916 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already out. 2021-07-26T06:08:33.927 INFO:tasks.workunit.client.0.smithi094.stdout:bringing up non_divergent 0 2 2021-07-26T06:08:33.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2021-07-26T06:08:33.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:08:34.284 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:08:34.285 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 0 0 0 60752 0 0 16 16 stale+active+clean 2021-07-26T06:07:49.053276+0000 29'106 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:08:34.285 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:08:34.286 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:08:34.287 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:08:34.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:34.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2021-07-26T06:08:34.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:08:34.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:08:34.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:08:34.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:08:34.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:08:34.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:08:34.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:08:34.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:08:34.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:08:34.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:08:34.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:08:34.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:08:34.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:08:34.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:08:34.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:08:34.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:08:34.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:08:34.310 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:08:34.310 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:08:34.311 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:08:34.311 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:08:34.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:08:34.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:08:34.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:08:34.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:08:34.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:08:34.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:08:34.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:08:34.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:08:34.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:08:34.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:08:34.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:08:34.316 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:08:34.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:08:34.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:08:34.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/0/whoami 2021-07-26T06:08:34.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:08:34.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:08:34.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:08:34.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:08:34.336 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:34.345+0000 7f963760df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:08:34.338 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:34.349+0000 7f963760df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:08:34.340 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:34.349+0000 7f963760df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:08:34.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:08:34.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:08:34.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:08:34.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:08:34.761 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:08:34.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:08:34.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:34.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:08:34.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:34.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:08:34.902 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:34.913+0000 7f963760df40 -1 Falling back to public interface 2021-07-26T06:08:35.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:08:36.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:08:36.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:36.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:08:36.192 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:08:36.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:36.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:08:36.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:08:37.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:08:37.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:37.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:08:37.595 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:08:37.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:37.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:08:37.804 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:37.813+0000 7f963760df40 -1 osd.0 29 log_to_monitors {default=true} 2021-07-26T06:08:37.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:08:38.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:08:38.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:38.980 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:08:38.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:08:38.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:38.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:08:39.353 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/38132,v1:127.0.0.1:6803/38132] [v2:127.0.0.1:6804/38132,v1:127.0.0.1:6805/38132] exists,up b02e7a44-f114-4850-90a6-03564e323691 2021-07-26T06:08:39.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:08:39.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:08:39.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:08:39.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:39.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2021-07-26T06:08:39.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:08:39.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:08:39.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:08:39.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:08:39.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:08:39.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:08:39.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:08:39.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:08:39.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:08:39.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:08:39.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:08:39.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:08:39.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:08:39.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:08:39.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:08:39.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:08:39.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:08:39.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:08:39.362 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:08:39.362 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:08:39.363 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:08:39.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:08:39.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:08:39.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:08:39.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:08:39.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:08:39.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:08:39.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:08:39.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:08:39.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:08:39.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:08:39.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:08:39.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:08:39.366 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:08:39.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:08:39.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/2/whoami 2021-07-26T06:08:39.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:08:39.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:08:39.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:08:39.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:08:39.395 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:39.405+0000 7fd34304af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:08:39.408 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:39.417+0000 7fd34304af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:08:39.410 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:39.421+0000 7fd34304af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:08:39.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:08:39.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:08:39.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:08:39.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:08:39.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:08:39.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:39.745 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:08:39.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:08:39.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:39.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:08:39.975 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:39.985+0000 7fd34304af40 -1 Falling back to public interface 2021-07-26T06:08:40.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:08:41.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:08:41.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:41.247 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:08:41.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:08:41.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:41.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:08:41.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:08:42.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:08:42.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:42.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:08:42.626 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:08:42.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:42.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:08:42.727 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:08:42.737+0000 7fd34304af40 -1 osd.2 29 log_to_monitors {default=true} 2021-07-26T06:08:43.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:08:44.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:08:44.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:08:44.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:08:44.069 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:08:44.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:08:44.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:08:44.465 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/38710,v1:127.0.0.1:6811/38710] [v2:127.0.0.1:6812/38710,v1:127.0.0.1:6813/38710] exists,up 4b82adfb-7a16-42ca-ab0d-8102aa9df430 2021-07-26T06:08:44.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:08:44.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:08:44.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:08:44.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:44.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2021-07-26T06:08:45.012 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already in. 2021-07-26T06:08:45.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:08:45.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2021-07-26T06:08:46.187 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already in. 2021-07-26T06:08:46.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:08:46.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2021-07-26T06:08:46.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:08:46.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:08:46.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:08:46.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:08:46.201 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:08:46.201 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:08:46.201 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:08:46.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:08:46.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:08:46.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:08:46.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:08:46.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:08:46.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:08:46.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:08:46.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:08:46.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:08:46.854 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:08:46.855 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:08:46.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:08:46.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:08:46.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:08:47.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=154618822660 2021-07-26T06:08:47.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 154618822660 2021-07-26T06:08:47.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-154618822660' 2021-07-26T06:08:47.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:08:47.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:08:47.126 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-07-26T06:08:47.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T06:08:47.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T06:08:47.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T06:08:47.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:08:47.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:08:47.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=163208757251 2021-07-26T06:08:47.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 163208757251 2021-07-26T06:08:47.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2021-07-26T06:08:47.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:08:47.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-154618822660 2021-07-26T06:08:47.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:08:47.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:08:47.287 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-154618822660 2021-07-26T06:08:47.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:08:47.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=154618822660 2021-07-26T06:08:47.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2021-07-26T06:08:47.289 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 154618822660 2021-07-26T06:08:47.290 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:08:47.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 154618822658 -lt 154618822660 2021-07-26T06:08:47.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:08:48.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:08:48.669 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:08:49.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 154618822661 -lt 154618822660 2021-07-26T06:08:49.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:08:49.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-163208757251 2021-07-26T06:08:49.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:08:49.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:08:49.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-163208757251 2021-07-26T06:08:49.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:08:49.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=163208757251 2021-07-26T06:08:49.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2021-07-26T06:08:49.047 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 163208757251 2021-07-26T06:08:49.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:08:49.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757252 -lt 163208757251 2021-07-26T06:08:49.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:08:49.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:49.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:49.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:08:49.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:08:49.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:08:49.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:08:49.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:08:49.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:08:49.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:08:49.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:08:50.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:08:50.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:08:50.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:50.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:50.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:08:50.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:08:50.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:08:50.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:08:50.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:08:50.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:08:50.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:08:50.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:08:50.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:08:50.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:08:50.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:08:50.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:08:50.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:08:50.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:08:51.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:08:51.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:08:51.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:51.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:51.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:08:51.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:08:51.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:08:51.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:08:51.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:08:51.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:08:51.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:08:51.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:08:51.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:08:52.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:08:52.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:08:52.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:08:52.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:08:52.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:08:52.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:08:52.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:08:52.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:08:52.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:08:52.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:08:52.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:08:52.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:08:52.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:08:52.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:08:52.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:08:52.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:52.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:53.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:08:53.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:08:53.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:08:53.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:08:53.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:08:53.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:08:53.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:08:53.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:08:53.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:08:53.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:08:53.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:08:53.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:08:53.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:08:53.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:08:54.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:08:54.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:08:54.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:08:54.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:08:54.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:08:54.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:08:54.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:08:54.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:08:54.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:08:54.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:08:54.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:54.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:54.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:08:54.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:08:54.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:08:54.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:08:54.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:08:54.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:08:54.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:08:54.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:08:54.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:08:55.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:08:55.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:08:55.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:08:55.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:08:55.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:08:56.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:08:56.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:08:56.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:08:56.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:08:56.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:08:56.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:08:56.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:08:56.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:08:56.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:08:56.597 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:08:56.597 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:56.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:08:57.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:08:57.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:08:57.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:08:57.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:08:57.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:08:57.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:08:57.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:08:57.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:08:57.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:08:57.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:08:57.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:08:57.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 4 >= 8 )) 2021-07-26T06:08:57.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:08:57.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 1.6 2021-07-26T06:08:59.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:08:59.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:08:59.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:08:59.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:08:59.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:08:59.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:08:59.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:08:59.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:08:59.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:08:59.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:08:59.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:08:59.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:00.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:00.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:00.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:00.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:00.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:00.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:00.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:00.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 5 >= 8 )) 2021-07-26T06:09:00.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:00.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 3.2 2021-07-26T06:09:03.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:03.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:03.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:03.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:03.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:03.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:03.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:03.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:04.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:04.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:04.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:04.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:04.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:04.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:04.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:04.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:04.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:04.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:04.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:04.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:04.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:05.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:05.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:05.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 6 >= 8 )) 2021-07-26T06:09:05.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:05.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 6.4 2021-07-26T06:09:11.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:11.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:11.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:11.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:11.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:11.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:11.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:11.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:11.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:11.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:11.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:11.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:12.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:12.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:12.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:12.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:12.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:12.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:12.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:12.281 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:12.282 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:12.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:12.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:12.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 7 >= 8 )) 2021-07-26T06:09:12.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:12.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 7.3 2021-07-26T06:09:20.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:20.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:20.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:20.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:20.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:20.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:20.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:20.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:20.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:20.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:20.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:20.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:21.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:21.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:21.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:21.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:21.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:21.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:21.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:21.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:21.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:21.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:21.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:21.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 8 >= 8 )) 2021-07-26T06:09:21.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph report 2021-07-26T06:09:22.035 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:09:22.035 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_fingerprint": "911f13a8-971e-4795-8d0c-76509363bcf1", 2021-07-26T06:09:22.035 INFO:tasks.workunit.client.0.smithi094.stdout: "version": "17.0.0-6358-g197e564f", 2021-07-26T06:09:22.036 INFO:tasks.workunit.client.0.smithi094.stdout: "commit": "197e564fdaee12596de1dc8f3253d1704ae66d5d", 2021-07-26T06:09:22.036 INFO:tasks.workunit.client.0.smithi094.stdout: "timestamp": "2021-07-26T06:09:22.047448+0000", 2021-07-26T06:09:22.036 INFO:tasks.workunit.client.0.smithi094.stdout: "tag": "", 2021-07-26T06:09:22.036 INFO:tasks.workunit.client.0.smithi094.stdout: "health": { 2021-07-26T06:09:22.037 INFO:tasks.workunit.client.0.smithi094.stdout: "status": "HEALTH_WARN", 2021-07-26T06:09:22.037 INFO:tasks.workunit.client.0.smithi094.stdout: "checks": { 2021-07-26T06:09:22.037 INFO:tasks.workunit.client.0.smithi094.stdout: "OSDMAP_FLAGS": { 2021-07-26T06:09:22.037 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:22.037 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:22.038 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "nodown,noout,noin flag(s) set", 2021-07-26T06:09:22.038 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 17 2021-07-26T06:09:22.038 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.038 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [], 2021-07-26T06:09:22.039 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:22.039 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.039 INFO:tasks.workunit.client.0.smithi094.stdout: "OSD_DOWN": { 2021-07-26T06:09:22.039 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:22.040 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:22.040 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 osds down", 2021-07-26T06:09:22.040 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:09:22.040 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.040 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:09:22.041 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.041 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "osd.1 (root=default,host=smithi094) is down" 2021-07-26T06:09:22.041 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.042 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.042 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:22.042 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.043 INFO:tasks.workunit.client.0.smithi094.stdout: "PG_DEGRADED": { 2021-07-26T06:09:22.043 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:22.043 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:22.043 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-07-26T06:09:22.043 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:09:22.044 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.044 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:09:22.044 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.044 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:09:22.045 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.045 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.045 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:22.045 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.045 INFO:tasks.workunit.client.0.smithi094.stdout: "POOL_APP_NOT_ENABLED": { 2021-07-26T06:09:22.046 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:22.046 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:22.046 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 pool(s) do not have an application enabled", 2021-07-26T06:09:22.046 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:09:22.046 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.047 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:09:22.047 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.047 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "application not enabled on pool 'test'" 2021-07-26T06:09:22.047 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.048 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.048 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-07-26T06:09:22.048 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.048 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.048 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:22.049 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.049 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.049 INFO:tasks.workunit.client.0.smithi094.stdout: "mutes": [] 2021-07-26T06:09:22.049 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.049 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_first_committed": 1, 2021-07-26T06:09:22.050 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_last_committed": 1, 2021-07-26T06:09:22.050 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap": { 2021-07-26T06:09:22.050 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:09:22.050 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:09:22.050 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:07:08.711461Z", 2021-07-26T06:09:22.051 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:07:08.711461Z", 2021-07-26T06:09:22.051 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release": 17, 2021-07-26T06:09:22.051 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release_name": "quincy", 2021-07-26T06:09:22.051 INFO:tasks.workunit.client.0.smithi094.stdout: "election_strategy": 1, 2021-07-26T06:09:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "disallowed_leaders: ": "", 2021-07-26T06:09:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": false, 2021-07-26T06:09:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "features": { 2021-07-26T06:09:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "persistent": [ 2021-07-26T06:09:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "kraken", 2021-07-26T06:09:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: "luminous", 2021-07-26T06:09:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: "mimic", 2021-07-26T06:09:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap-prune", 2021-07-26T06:09:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: "nautilus", 2021-07-26T06:09:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: "octopus", 2021-07-26T06:09:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: "pacific", 2021-07-26T06:09:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: "elector-pinging", 2021-07-26T06:09:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: "quincy" 2021-07-26T06:09:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: "optional": [] 2021-07-26T06:09:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: "mons": [ 2021-07-26T06:09:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: "rank": 0, 2021-07-26T06:09:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "a", 2021-07-26T06:09:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.058 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.058 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115", 2021-07-26T06:09:22.058 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 0 2021-07-26T06:09:22.058 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.058 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.059 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.059 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115/0", 2021-07-26T06:09:22.059 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:7115/0", 2021-07-26T06:09:22.059 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 0, 2021-07-26T06:09:22.060 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 0, 2021-07-26T06:09:22.060 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_location": "{}" 2021-07-26T06:09:22.060 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.060 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.060 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.061 INFO:tasks.workunit.client.0.smithi094.stdout: "quorum": [ 2021-07-26T06:09:22.061 INFO:tasks.workunit.client.0.smithi094.stdout: 0 2021-07-26T06:09:22.061 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.061 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap": { 2021-07-26T06:09:22.062 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 43, 2021-07-26T06:09:22.062 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:09:22.062 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:07:09.635149+0000", 2021-07-26T06:09:22.062 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:08:46.267898+0000", 2021-07-26T06:09:22.062 INFO:tasks.workunit.client.0.smithi094.stdout: "last_up_change": "2021-07-26T06:08:43.043459+0000", 2021-07-26T06:09:22.063 INFO:tasks.workunit.client.0.smithi094.stdout: "last_in_change": "2021-07-26T06:08:45.598209+0000", 2021-07-26T06:09:22.063 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-07-26T06:09:22.063 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_num": 5800384, 2021-07-26T06:09:22.063 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_set": [ 2021-07-26T06:09:22.063 INFO:tasks.workunit.client.0.smithi094.stdout: "nodown", 2021-07-26T06:09:22.064 INFO:tasks.workunit.client.0.smithi094.stdout: "noin", 2021-07-26T06:09:22.064 INFO:tasks.workunit.client.0.smithi094.stdout: "noout", 2021-07-26T06:09:22.064 INFO:tasks.workunit.client.0.smithi094.stdout: "pglog_hardlimit", 2021-07-26T06:09:22.064 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snapdirs", 2021-07-26T06:09:22.064 INFO:tasks.workunit.client.0.smithi094.stdout: "recovery_deletes", 2021-07-26T06:09:22.065 INFO:tasks.workunit.client.0.smithi094.stdout: "sortbitwise" 2021-07-26T06:09:22.065 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.065 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_version": 9, 2021-07-26T06:09:22.065 INFO:tasks.workunit.client.0.smithi094.stdout: "full_ratio": 0.99000000953674316, 2021-07-26T06:09:22.066 INFO:tasks.workunit.client.0.smithi094.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-07-26T06:09:22.066 INFO:tasks.workunit.client.0.smithi094.stdout: "nearfull_ratio": 0.85000002384185791, 2021-07-26T06:09:22.066 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_snapshot": "", 2021-07-26T06:09:22.066 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_max": 1, 2021-07-26T06:09:22.066 INFO:tasks.workunit.client.0.smithi094.stdout: "max_osd": 3, 2021-07-26T06:09:22.067 INFO:tasks.workunit.client.0.smithi094.stdout: "require_min_compat_client": "luminous", 2021-07-26T06:09:22.067 INFO:tasks.workunit.client.0.smithi094.stdout: "min_compat_client": "jewel", 2021-07-26T06:09:22.067 INFO:tasks.workunit.client.0.smithi094.stdout: "require_osd_release": "quincy", 2021-07-26T06:09:22.067 INFO:tasks.workunit.client.0.smithi094.stdout: "pools": [ 2021-07-26T06:09:22.068 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.068 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:09:22.068 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_name": "test", 2021-07-26T06:09:22.068 INFO:tasks.workunit.client.0.smithi094.stdout: "create_time": "2021-07-26T06:07:48.003331+0000", 2021-07-26T06:09:22.068 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": 1, 2021-07-26T06:09:22.069 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_names": "hashpspool", 2021-07-26T06:09:22.069 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:09:22.069 INFO:tasks.workunit.client.0.smithi094.stdout: "size": 3, 2021-07-26T06:09:22.069 INFO:tasks.workunit.client.0.smithi094.stdout: "min_size": 2, 2021-07-26T06:09:22.069 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_rule": 0, 2021-07-26T06:09:22.070 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_count": 0, 2021-07-26T06:09:22.070 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_target": 0, 2021-07-26T06:09:22.070 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_barrier": 0, 2021-07-26T06:09:22.070 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-07-26T06:09:22.070 INFO:tasks.workunit.client.0.smithi094.stdout: "object_hash": 2, 2021-07-26T06:09:22.071 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_autoscale_mode": "off", 2021-07-26T06:09:22.071 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num": 1, 2021-07-26T06:09:22.071 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num": 1, 2021-07-26T06:09:22.071 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num_target": 1, 2021-07-26T06:09:22.072 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_target": 1, 2021-07-26T06:09:22.072 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_pending": 1, 2021-07-26T06:09:22.072 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pg_merge_meta": { 2021-07-26T06:09:22.072 INFO:tasks.workunit.client.0.smithi094.stdout: "source_pgid": "0.0", 2021-07-26T06:09:22.072 INFO:tasks.workunit.client.0.smithi094.stdout: "ready_epoch": 0, 2021-07-26T06:09:22.073 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_started": 0, 2021-07-26T06:09:22.073 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": 0, 2021-07-26T06:09:22.073 INFO:tasks.workunit.client.0.smithi094.stdout: "source_version": "0'0", 2021-07-26T06:09:22.073 INFO:tasks.workunit.client.0.smithi094.stdout: "target_version": "0'0" 2021-07-26T06:09:22.074 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.074 INFO:tasks.workunit.client.0.smithi094.stdout: "last_change": "29", 2021-07-26T06:09:22.074 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend": "0", 2021-07-26T06:09:22.074 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_prenautilus": "0", 2021-07-26T06:09:22.074 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_preluminous": "0", 2021-07-26T06:09:22.075 INFO:tasks.workunit.client.0.smithi094.stdout: "auid": 0, 2021-07-26T06:09:22.075 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_mode": "selfmanaged", 2021-07-26T06:09:22.075 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_seq": 0, 2021-07-26T06:09:22.075 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_epoch": 0, 2021-07-26T06:09:22.075 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_snaps": [], 2021-07-26T06:09:22.076 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps": "[]", 2021-07-26T06:09:22.076 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_bytes": 0, 2021-07-26T06:09:22.076 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_objects": 0, 2021-07-26T06:09:22.076 INFO:tasks.workunit.client.0.smithi094.stdout: "tiers": [], 2021-07-26T06:09:22.076 INFO:tasks.workunit.client.0.smithi094.stdout: "tier_of": -1, 2021-07-26T06:09:22.077 INFO:tasks.workunit.client.0.smithi094.stdout: "read_tier": -1, 2021-07-26T06:09:22.077 INFO:tasks.workunit.client.0.smithi094.stdout: "write_tier": -1, 2021-07-26T06:09:22.077 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_mode": "none", 2021-07-26T06:09:22.077 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_bytes": 0, 2021-07-26T06:09:22.077 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_objects": 0, 2021-07-26T06:09:22.078 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-07-26T06:09:22.078 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-07-26T06:09:22.078 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_full_ratio_micro": 800000, 2021-07-26T06:09:22.078 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_flush_age": 0, 2021-07-26T06:09:22.079 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_evict_age": 0, 2021-07-26T06:09:22.079 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profile": "", 2021-07-26T06:09:22.079 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_params": { 2021-07-26T06:09:22.079 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "none" 2021-07-26T06:09:22.079 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.080 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_period": 0, 2021-07-26T06:09:22.080 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_count": 0, 2021-07-26T06:09:22.080 INFO:tasks.workunit.client.0.smithi094.stdout: "use_gmt_hitset": true, 2021-07-26T06:09:22.080 INFO:tasks.workunit.client.0.smithi094.stdout: "min_read_recency_for_promote": 0, 2021-07-26T06:09:22.081 INFO:tasks.workunit.client.0.smithi094.stdout: "min_write_recency_for_promote": 0, 2021-07-26T06:09:22.081 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_grade_decay_rate": 0, 2021-07-26T06:09:22.081 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_search_last_n": 0, 2021-07-26T06:09:22.081 INFO:tasks.workunit.client.0.smithi094.stdout: "grade_table": [], 2021-07-26T06:09:22.081 INFO:tasks.workunit.client.0.smithi094.stdout: "stripe_width": 0, 2021-07-26T06:09:22.082 INFO:tasks.workunit.client.0.smithi094.stdout: "expected_num_objects": 0, 2021-07-26T06:09:22.082 INFO:tasks.workunit.client.0.smithi094.stdout: "fast_read": false, 2021-07-26T06:09:22.082 INFO:tasks.workunit.client.0.smithi094.stdout: "options": {}, 2021-07-26T06:09:22.082 INFO:tasks.workunit.client.0.smithi094.stdout: "application_metadata": {} 2021-07-26T06:09:22.082 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.083 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.083 INFO:tasks.workunit.client.0.smithi094.stdout: "osds": [ 2021-07-26T06:09:22.083 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.083 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:22.084 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "b02e7a44-f114-4850-90a6-03564e323691", 2021-07-26T06:09:22.084 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:09:22.084 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:09:22.084 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:09:22.084 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:09:22.085 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 5, 2021-07-26T06:09:22.085 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:09:22.085 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 36, 2021-07-26T06:09:22.085 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 41, 2021-07-26T06:09:22.085 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 30, 2021-07-26T06:09:22.086 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:09:22.086 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:22.086 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.086 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.086 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.087 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6802", 2021-07-26T06:09:22.087 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.087 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.087 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.088 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.088 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6803", 2021-07-26T06:09:22.088 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.088 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.088 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.089 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.089 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:09:22.089 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.089 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.089 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.090 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6804", 2021-07-26T06:09:22.090 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.090 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.090 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.091 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.091 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6805", 2021-07-26T06:09:22.091 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.091 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.091 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.092 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.092 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:09:22.092 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.092 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.092 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.093 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6808", 2021-07-26T06:09:22.093 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.093 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.093 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.093 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.093 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6809", 2021-07-26T06:09:22.094 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.094 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.094 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.094 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.095 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:09:22.095 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.095 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.095 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.095 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6806", 2021-07-26T06:09:22.096 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.096 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.096 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.096 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.096 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6807", 2021-07-26T06:09:22.097 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:22.097 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.097 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.097 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.098 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6803/38132", 2021-07-26T06:09:22.098 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6805/38132", 2021-07-26T06:09:22.098 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6809/38132", 2021-07-26T06:09:22.098 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6807/38132", 2021-07-26T06:09:22.099 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:09:22.099 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:09:22.099 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:09:22.099 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.099 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.100 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.100 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:22.100 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "a4d2525d-0097-4f8c-b13e-ee3a5e599d6d", 2021-07-26T06:09:22.100 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 0, 2021-07-26T06:09:22.100 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:09:22.101 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:09:22.101 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:09:22.101 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 0, 2021-07-26T06:09:22.101 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 0, 2021-07-26T06:09:22.102 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 10, 2021-07-26T06:09:22.102 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 23, 2021-07-26T06:09:22.102 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 31, 2021-07-26T06:09:22.102 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:09:22.102 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:22.103 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.103 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.103 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.103 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:09:22.104 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.104 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.104 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.104 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.104 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:09:22.105 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.105 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.105 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.105 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.106 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:09:22.106 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.106 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.106 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.106 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:09:22.107 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.107 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.107 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.107 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.107 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:09:22.108 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.108 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.108 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.108 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.108 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:09:22.109 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.109 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.109 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.109 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:09:22.110 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.110 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.110 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.110 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.110 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:09:22.111 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.111 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.111 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.111 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.112 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:09:22.112 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.112 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.112 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.113 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:09:22.113 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.113 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.113 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.113 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.114 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:09:22.114 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:22.114 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.114 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.115 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.115 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/32635", 2021-07-26T06:09:22.115 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/32635", 2021-07-26T06:09:22.115 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/32635", 2021-07-26T06:09:22.115 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/32635", 2021-07-26T06:09:22.116 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:09:22.116 INFO:tasks.workunit.client.0.smithi094.stdout: "exists" 2021-07-26T06:09:22.116 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.116 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.117 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.117 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:22.117 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "4b82adfb-7a16-42ca-ab0d-8102aa9df430", 2021-07-26T06:09:22.117 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:09:22.117 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:09:22.117 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:09:22.118 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:09:22.118 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 15, 2021-07-26T06:09:22.118 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:09:22.118 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 38, 2021-07-26T06:09:22.119 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 0, 2021-07-26T06:09:22.119 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 32, 2021-07-26T06:09:22.119 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:09:22.119 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:22.119 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.120 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.120 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.120 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:09:22.120 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.120 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.121 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.121 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.121 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:09:22.121 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.122 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.122 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.122 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.122 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:09:22.123 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.123 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.123 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.123 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:09:22.123 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.124 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.124 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.124 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.124 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:09:22.125 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.125 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.125 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.125 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.125 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:09:22.126 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.126 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.126 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.126 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:09:22.126 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.127 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.127 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.127 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.127 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:09:22.128 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.128 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.128 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.128 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.128 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:09:22.129 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:22.129 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.129 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:22.129 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:09:22.129 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.130 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.130 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.130 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:22.130 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:09:22.131 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:22.131 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.131 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.131 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.131 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/38710", 2021-07-26T06:09:22.132 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/38710", 2021-07-26T06:09:22.132 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/38710", 2021-07-26T06:09:22.132 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/38710", 2021-07-26T06:09:22.132 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:09:22.132 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:09:22.133 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:09:22.133 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.133 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.133 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.134 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_xinfo": [ 2021-07-26T06:09:22.134 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.134 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:22.134 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:08:31.382366+0000", 2021-07-26T06:09:22.134 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:09:22.135 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:09:22.135 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:09:22.135 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:09:22.135 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:07:21.271225+0000", 2021-07-26T06:09:22.135 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:09:22.136 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.136 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.136 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:22.136 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:08:31.967926+0000", 2021-07-26T06:09:22.136 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:09:22.137 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:09:22.137 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:09:22.137 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:09:22.137 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:07:33.240355+0000", 2021-07-26T06:09:22.138 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:09:22.138 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.138 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.138 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:22.138 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:08:32.551830+0000", 2021-07-26T06:09:22.139 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:09:22.139 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:09:22.139 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:09:22.139 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:09:22.139 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:07:44.587495+0000", 2021-07-26T06:09:22.140 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:09:22.140 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.140 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.140 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap": [], 2021-07-26T06:09:22.141 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap_items": [], 2021-07-26T06:09:22.141 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_temp": [], 2021-07-26T06:09:22.141 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_temp": [], 2021-07-26T06:09:22.141 INFO:tasks.workunit.client.0.smithi094.stdout: "blocklist": {}, 2021-07-26T06:09:22.141 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profiles": { 2021-07-26T06:09:22.142 INFO:tasks.workunit.client.0.smithi094.stdout: "default": { 2021-07-26T06:09:22.142 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:09:22.142 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "2", 2021-07-26T06:09:22.142 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:09:22.142 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van" 2021-07-26T06:09:22.143 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.143 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.143 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps_queue": [], 2021-07-26T06:09:22.143 INFO:tasks.workunit.client.0.smithi094.stdout: "new_removed_snaps": [], 2021-07-26T06:09:22.143 INFO:tasks.workunit.client.0.smithi094.stdout: "new_purged_snaps": [], 2021-07-26T06:09:22.144 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_node_flags": {}, 2021-07-26T06:09:22.144 INFO:tasks.workunit.client.0.smithi094.stdout: "device_class_flags": {}, 2021-07-26T06:09:22.144 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": { 2021-07-26T06:09:22.144 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_enabled": false, 2021-07-26T06:09:22.145 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_bucket_count": 0, 2021-07-26T06:09:22.145 INFO:tasks.workunit.client.0.smithi094.stdout: "degraded_stretch_mode": 0, 2021-07-26T06:09:22.145 INFO:tasks.workunit.client.0.smithi094.stdout: "recovering_stretch_mode": 0, 2021-07-26T06:09:22.145 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_bucket": 0 2021-07-26T06:09:22.145 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.146 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.146 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_metadata": [ 2021-07-26T06:09:22.146 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.146 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:22.146 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:09:22.147 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6804/38132,v1:127.0.0.1:6805/38132]", 2021-07-26T06:09:22.147 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:09:22.147 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:09:22.147 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:09:22.147 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:09:22.148 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:09:22.148 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:09:22.148 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:09:22.148 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:09:22.149 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:09:22.149 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-07-26T06:09:22.149 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:09:22.149 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:09:22.149 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:09:22.150 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:09:22.150 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:09:22.150 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:09:22.150 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:09:22.150 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:09:22.151 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:09:22.151 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:09:22.151 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:09:22.151 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:09:22.151 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:09:22.152 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:09:22.152 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:09:22.152 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6802/38132,v1:127.0.0.1:6803/38132]", 2021-07-26T06:09:22.152 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:09:22.152 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/38132,v1:127.0.0.1:6809/38132]", 2021-07-26T06:09:22.153 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/38132,v1:127.0.0.1:6807/38132]", 2021-07-26T06:09:22.153 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:09:22.153 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:09:22.153 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:09:22.153 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:09:22.154 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:09:22.154 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:09:22.154 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:09:22.154 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:09:22.155 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:09:22.155 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/0", 2021-07-26T06:09:22.155 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:09:22.155 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:09:22.155 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:09:22.156 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.156 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.156 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:22.156 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:09:22.156 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/32635,v1:127.0.0.1:6813/32635]", 2021-07-26T06:09:22.157 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:09:22.157 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:09:22.157 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:09:22.157 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:09:22.157 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:09:22.158 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:09:22.158 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:09:22.158 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:09:22.158 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:09:22.158 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-07-26T06:09:22.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:09:22.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:09:22.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:09:22.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:09:22.159 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:09:22.160 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:09:22.160 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:09:22.160 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:09:22.160 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:09:22.160 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:09:22.161 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:09:22.161 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:09:22.161 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:09:22.161 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:09:22.162 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:09:22.162 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/32635,v1:127.0.0.1:6811/32635]", 2021-07-26T06:09:22.162 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:09:22.162 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/32635,v1:127.0.0.1:6817/32635]", 2021-07-26T06:09:22.162 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/32635,v1:127.0.0.1:6815/32635]", 2021-07-26T06:09:22.163 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:09:22.163 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:09:22.163 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:09:22.163 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:09:22.163 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:09:22.164 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:09:22.164 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:09:22.164 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:09:22.164 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:09:22.164 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/1", 2021-07-26T06:09:22.165 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:09:22.165 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:09:22.165 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:09:22.165 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.165 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:09:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/38710,v1:127.0.0.1:6813/38710]", 2021-07-26T06:09:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:09:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:09:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:09:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:09:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:09:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:09:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:09:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:09:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:09:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-07-26T06:09:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:09:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:09:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:09:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:09:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:09:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:09:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:09:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:09:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:09:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:09:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:09:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:09:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:09:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:09:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:09:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/38710,v1:127.0.0.1:6811/38710]", 2021-07-26T06:09:22.172 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:09:22.172 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/38710,v1:127.0.0.1:6817/38710]", 2021-07-26T06:09:22.172 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/38710,v1:127.0.0.1:6815/38710]", 2021-07-26T06:09:22.172 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:09:22.172 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:09:22.173 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:09:22.173 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:09:22.173 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:09:22.173 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:09:22.173 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:09:22.174 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:09:22.174 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:09:22.174 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/2", 2021-07-26T06:09:22.174 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:09:22.175 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:09:22.175 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:09:22.175 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.175 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.175 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_clean_epochs": { 2021-07-26T06:09:22.176 INFO:tasks.workunit.client.0.smithi094.stdout: "min_last_epoch_clean": 0, 2021-07-26T06:09:22.176 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": { 2021-07-26T06:09:22.176 INFO:tasks.workunit.client.0.smithi094.stdout: "per_pool": [] 2021-07-26T06:09:22.176 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.176 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_epochs": [ 2021-07-26T06:09:22.177 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.177 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:22.177 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 36 2021-07-26T06:09:22.177 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.177 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.178 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:22.178 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 10 2021-07-26T06:09:22.178 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.178 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.178 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:22.179 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 38 2021-07-26T06:09:22.179 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.179 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.179 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.179 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_first_committed": 1, 2021-07-26T06:09:22.180 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_last_committed": 43, 2021-07-26T06:09:22.180 INFO:tasks.workunit.client.0.smithi094.stdout: "crushmap": { 2021-07-26T06:09:22.180 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": [ 2021-07-26T06:09:22.180 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.180 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:22.181 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.0", 2021-07-26T06:09:22.181 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:09:22.181 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.181 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.182 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:22.182 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.1", 2021-07-26T06:09:22.182 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:09:22.182 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.182 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.183 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:22.183 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.2", 2021-07-26T06:09:22.183 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:09:22.183 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.183 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.184 INFO:tasks.workunit.client.0.smithi094.stdout: "types": [ 2021-07-26T06:09:22.184 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.184 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 0, 2021-07-26T06:09:22.184 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd" 2021-07-26T06:09:22.184 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.185 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.185 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:09:22.185 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "host" 2021-07-26T06:09:22.185 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.185 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.186 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 2, 2021-07-26T06:09:22.186 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "chassis" 2021-07-26T06:09:22.186 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.186 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.186 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 3, 2021-07-26T06:09:22.187 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "rack" 2021-07-26T06:09:22.187 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.187 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.187 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 4, 2021-07-26T06:09:22.188 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "row" 2021-07-26T06:09:22.188 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.188 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.188 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 5, 2021-07-26T06:09:22.188 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pdu" 2021-07-26T06:09:22.189 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.189 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.189 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 6, 2021-07-26T06:09:22.189 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pod" 2021-07-26T06:09:22.189 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.190 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.190 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 7, 2021-07-26T06:09:22.190 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "room" 2021-07-26T06:09:22.190 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.191 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.191 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 8, 2021-07-26T06:09:22.191 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "datacenter" 2021-07-26T06:09:22.191 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.191 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.192 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 9, 2021-07-26T06:09:22.192 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "zone" 2021-07-26T06:09:22.192 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.192 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.192 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 10, 2021-07-26T06:09:22.193 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "region" 2021-07-26T06:09:22.193 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.193 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.193 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:09:22.194 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "root" 2021-07-26T06:09:22.194 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.194 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.194 INFO:tasks.workunit.client.0.smithi094.stdout: "buckets": [ 2021-07-26T06:09:22.194 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.195 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -1, 2021-07-26T06:09:22.195 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default", 2021-07-26T06:09:22.195 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:09:22.195 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:09:22.195 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:22.196 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:22.196 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:22.196 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:22.196 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.196 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:09:22.197 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:22.197 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:22.197 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.197 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.198 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.198 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.198 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -2, 2021-07-26T06:09:22.198 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default~hdd", 2021-07-26T06:09:22.198 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:09:22.199 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:09:22.199 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:22.199 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:22.199 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:22.199 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:22.200 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.200 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:09:22.200 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:22.200 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:22.200 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.201 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.201 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.201 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.201 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:09:22.202 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094", 2021-07-26T06:09:22.202 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:09:22.202 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:09:22.202 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:22.202 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:22.203 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:22.203 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:22.203 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.203 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:22.203 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:22.204 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:22.204 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.204 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.204 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:22.205 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:22.205 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:09:22.205 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.205 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.205 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:22.206 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:22.206 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:09:22.206 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.206 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.206 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.207 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.207 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:09:22.207 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094~hdd", 2021-07-26T06:09:22.207 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:09:22.207 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:09:22.208 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:22.208 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:22.208 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:22.208 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:22.208 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.209 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:22.209 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:22.209 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:22.209 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.209 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.210 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:22.210 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:22.210 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:09:22.210 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.211 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.211 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:22.211 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:22.211 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:09:22.211 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.212 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.212 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.212 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.212 INFO:tasks.workunit.client.0.smithi094.stdout: "rules": [ 2021-07-26T06:09:22.212 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.213 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 0, 2021-07-26T06:09:22.213 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "replicated_rule", 2021-07-26T06:09:22.213 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:09:22.213 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:09:22.214 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.214 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:09:22.214 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:09:22.214 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:09:22.214 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.215 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.215 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_firstn", 2021-07-26T06:09:22.215 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:09:22.215 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:09:22.215 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.216 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.216 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:09:22.216 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.216 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:22.217 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.217 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.217 INFO:tasks.workunit.client.0.smithi094.stdout: "tunables": { 2021-07-26T06:09:22.217 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_tries": 0, 2021-07-26T06:09:22.217 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_fallback_tries": 0, 2021-07-26T06:09:22.218 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_total_tries": 50, 2021-07-26T06:09:22.218 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_descend_once": 1, 2021-07-26T06:09:22.218 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_vary_r": 1, 2021-07-26T06:09:22.218 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_stable": 1, 2021-07-26T06:09:22.219 INFO:tasks.workunit.client.0.smithi094.stdout: "straw_calc_version": 1, 2021-07-26T06:09:22.219 INFO:tasks.workunit.client.0.smithi094.stdout: "allowed_bucket_algs": 54, 2021-07-26T06:09:22.219 INFO:tasks.workunit.client.0.smithi094.stdout: "profile": "jewel", 2021-07-26T06:09:22.219 INFO:tasks.workunit.client.0.smithi094.stdout: "optimal_tunables": 1, 2021-07-26T06:09:22.219 INFO:tasks.workunit.client.0.smithi094.stdout: "legacy_tunables": 0, 2021-07-26T06:09:22.220 INFO:tasks.workunit.client.0.smithi094.stdout: "minimum_required_version": "jewel", 2021-07-26T06:09:22.220 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables": 1, 2021-07-26T06:09:22.220 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables2": 1, 2021-07-26T06:09:22.220 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v2_rules": 0, 2021-07-26T06:09:22.220 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables3": 1, 2021-07-26T06:09:22.221 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v3_rules": 0, 2021-07-26T06:09:22.221 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v4_buckets": 1, 2021-07-26T06:09:22.221 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables5": 1, 2021-07-26T06:09:22.221 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v5_rules": 0 2021-07-26T06:09:22.222 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.222 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_args": {} 2021-07-26T06:09:22.222 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.222 INFO:tasks.workunit.client.0.smithi094.stdout: "fsmap": { 2021-07-26T06:09:22.222 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:09:22.223 INFO:tasks.workunit.client.0.smithi094.stdout: "default_fscid": -1, 2021-07-26T06:09:22.223 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": { 2021-07-26T06:09:22.223 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": {}, 2021-07-26T06:09:22.223 INFO:tasks.workunit.client.0.smithi094.stdout: "ro_compat": {}, 2021-07-26T06:09:22.223 INFO:tasks.workunit.client.0.smithi094.stdout: "incompat": { 2021-07-26T06:09:22.224 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_1": "base v0.20", 2021-07-26T06:09:22.224 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_2": "client writeable ranges", 2021-07-26T06:09:22.224 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_3": "default file layouts on dirs", 2021-07-26T06:09:22.224 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_4": "dir inode in separate object", 2021-07-26T06:09:22.225 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_5": "mds uses versioned encoding", 2021-07-26T06:09:22.225 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_6": "dirfrag is stored in omap", 2021-07-26T06:09:22.225 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_8": "no anchor table", 2021-07-26T06:09:22.225 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_9": "file layout v2", 2021-07-26T06:09:22.225 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_10": "snaprealm v2" 2021-07-26T06:09:22.226 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.226 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.226 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_flags": { 2021-07-26T06:09:22.226 INFO:tasks.workunit.client.0.smithi094.stdout: "enable_multiple": true, 2021-07-26T06:09:22.226 INFO:tasks.workunit.client.0.smithi094.stdout: "ever_enabled_multiple": true 2021-07-26T06:09:22.227 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.227 INFO:tasks.workunit.client.0.smithi094.stdout: "standbys": [], 2021-07-26T06:09:22.227 INFO:tasks.workunit.client.0.smithi094.stdout: "filesystems": [] 2021-07-26T06:09:22.227 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.227 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_first_committed": 1, 2021-07-26T06:09:22.228 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_last_committed": 1, 2021-07-26T06:09:22.228 INFO:tasks.workunit.client.0.smithi094.stdout: "auth": { 2021-07-26T06:09:22.228 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:09:22.228 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 5, 2021-07-26T06:09:22.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_secrets": 3 2021-07-26T06:09:22.229 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:09:22.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_active": 1, 2021-07-26T06:09:22.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_unknown": 0, 2021-07-26T06:09:22.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osd": 3, 2021-07-26T06:09:22.230 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_sum": { 2021-07-26T06:09:22.230 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:09:22.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61700, 2021-07-26T06:09:22.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 100, 2021-07-26T06:09:22.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:09:22.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 300, 2021-07-26T06:09:22.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:09:22.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:09:22.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 100, 2021-07-26T06:09:22.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:09:22.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:09:22.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 100, 2021-07-26T06:09:22.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:09:22.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:09:22.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:09:22.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 100, 2021-07-26T06:09:22.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 100, 2021-07-26T06:09:22.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:09:22.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:09:22.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:09:22.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:09:22.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:09:22.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:09:22.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:09:22.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:09:22.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:09:22.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:09:22.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:09:22.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:09:22.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:09:22.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:09:22.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:09:22.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:09:22.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:09:22.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:09:22.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:09:22.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:09:22.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:09:22.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:09:22.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:09:22.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:09:22.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:09:22.238 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.239 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:09:22.239 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:09:22.239 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:09:22.239 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:22.240 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 0, 2021-07-26T06:09:22.240 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 0, 2021-07-26T06:09:22.240 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:22.240 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:22.240 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:22.240 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:09:22.241 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:09:22.241 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.241 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 16, 2021-07-26T06:09:22.241 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 16, 2021-07-26T06:09:22.241 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 2, 2021-07-26T06:09:22.242 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 2, 2021-07-26T06:09:22.242 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 0 2021-07-26T06:09:22.242 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.242 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum": { 2021-07-26T06:09:22.242 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:09:22.243 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:09:22.243 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:09:22.243 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:09:22.243 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:09:22.243 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:09:22.244 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:09:22.244 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18424, 2021-07-26T06:09:22.244 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2136, 2021-07-26T06:09:22.244 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:09:22.244 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16253, 2021-07-26T06:09:22.244 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554376, 2021-07-26T06:09:22.245 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:09:22.245 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:09:22.245 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103681024, 2021-07-26T06:09:22.245 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:22.245 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2187264, 2021-07-26T06:09:22.246 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 468594, 2021-07-26T06:09:22.246 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:22.246 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:22.246 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:22.247 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:09:22.247 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16643984 2021-07-26T06:09:22.247 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.247 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:09:22.247 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:09:22.247 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:09:22.248 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:09:22.248 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:09:22.248 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:09:22.248 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:09:22.248 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.249 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:09:22.249 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:09:22.249 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:09:22.249 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:09:22.249 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:09:22.250 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.250 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:09:22.250 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:09:22.250 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.250 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum_by_class": { 2021-07-26T06:09:22.251 INFO:tasks.workunit.client.0.smithi094.stdout: "hdd": { 2021-07-26T06:09:22.251 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:09:22.251 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:09:22.251 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:09:22.251 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:09:22.252 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:09:22.252 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:09:22.252 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:09:22.252 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18424, 2021-07-26T06:09:22.252 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2136, 2021-07-26T06:09:22.253 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:09:22.253 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16253, 2021-07-26T06:09:22.253 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554376, 2021-07-26T06:09:22.253 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:09:22.254 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:09:22.254 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103681024, 2021-07-26T06:09:22.254 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:22.254 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2187264, 2021-07-26T06:09:22.254 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 468594, 2021-07-26T06:09:22.254 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:22.255 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:22.255 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:22.255 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:09:22.255 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16643984 2021-07-26T06:09:22.255 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.256 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:09:22.256 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:09:22.256 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:09:22.256 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:09:22.256 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:09:22.257 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:09:22.257 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:09:22.257 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.257 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:09:22.257 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:09:22.258 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:09:22.258 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:09:22.258 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:09:22.258 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.258 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:09:22.259 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:09:22.259 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.259 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.259 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_stats": [ 2021-07-26T06:09:22.259 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.260 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:09:22.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:09:22.260 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:09:22.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61700, 2021-07-26T06:09:22.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 100, 2021-07-26T06:09:22.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:09:22.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 300, 2021-07-26T06:09:22.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:09:22.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:09:22.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 100, 2021-07-26T06:09:22.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:09:22.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:09:22.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 100, 2021-07-26T06:09:22.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:09:22.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:09:22.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:09:22.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 100, 2021-07-26T06:09:22.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 100, 2021-07-26T06:09:22.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:09:22.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:09:22.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:09:22.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:09:22.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:09:22.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:09:22.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:09:22.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:09:22.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:09:22.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:09:22.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:09:22.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:09:22.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:09:22.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:09:22.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:09:22.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:09:22.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:09:22.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:09:22.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:09:22.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:09:22.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:09:22.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:09:22.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:09:22.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:09:22.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:09:22.269 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.269 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:09:22.269 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:09:22.269 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:09:22.269 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:22.270 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 1220608, 2021-07-26T06:09:22.270 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 184152, 2021-07-26T06:09:22.270 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:22.270 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:22.271 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:22.271 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:09:22.271 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:09:22.271 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.271 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 16, 2021-07-26T06:09:22.272 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 16, 2021-07-26T06:09:22.272 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 2, 2021-07-26T06:09:22.272 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 2, 2021-07-26T06:09:22.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 3 2021-07-26T06:09:22.272 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.273 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.273 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_stats": [ 2021-07-26T06:09:22.273 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.273 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:22.273 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 154618822667 2021-07-26T06:09:22.274 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.274 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.274 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:22.274 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 42949672978 2021-07-26T06:09:22.274 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.275 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.275 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:22.275 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 163208757258 2021-07-26T06:09:22.275 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.275 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.276 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_state": [ 2021-07-26T06:09:22.276 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.276 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized+degraded", 2021-07-26T06:09:22.276 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 1 2021-07-26T06:09:22.276 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.277 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.277 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_osd": [ 2021-07-26T06:09:22.277 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.277 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:22.277 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:09:22.278 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:09:22.278 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:09:22.278 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.278 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.278 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:22.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 1, 2021-07-26T06:09:22.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:09:22.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:09:22.279 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.280 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.280 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:22.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:09:22.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 0, 2021-07-26T06:09:22.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:09:22.280 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.281 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.281 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": [ 2021-07-26T06:09:22.281 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:22.281 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:09:22.282 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": {} 2021-07-26T06:09:22.282 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.282 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:22.282 INFO:tasks.workunit.client.0.smithi094.stdout: "servicemap": { 2021-07-26T06:09:22.282 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 4, 2021-07-26T06:09:22.283 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:08:34.363306+0000", 2021-07-26T06:09:22.283 INFO:tasks.workunit.client.0.smithi094.stdout: "services": { 2021-07-26T06:09:22.283 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": { 2021-07-26T06:09:22.283 INFO:tasks.workunit.client.0.smithi094.stdout: "daemons": { 2021-07-26T06:09:22.283 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": "", 2021-07-26T06:09:22.284 INFO:tasks.workunit.client.0.smithi094.stdout: "2": { 2021-07-26T06:09:22.284 INFO:tasks.workunit.client.0.smithi094.stdout: "start_epoch": 0, 2021-07-26T06:09:22.284 INFO:tasks.workunit.client.0.smithi094.stdout: "start_stamp": "0.000000", 2021-07-26T06:09:22.284 INFO:tasks.workunit.client.0.smithi094.stdout: "gid": 0, 2021-07-26T06:09:22.284 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "(unrecognized address family 0)/0", 2021-07-26T06:09:22.285 INFO:tasks.workunit.client.0.smithi094.stdout: "metadata": {}, 2021-07-26T06:09:22.285 INFO:tasks.workunit.client.0.smithi094.stdout: "task_status": {} 2021-07-26T06:09:22.285 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.285 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.285 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.286 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.286 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:22.286 INFO:tasks.workunit.client.0.smithi094.stdout: "paxos": { 2021-07-26T06:09:22.286 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:09:22.286 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 387, 2021-07-26T06:09:22.287 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pn": 0, 2021-07-26T06:09:22.287 INFO:tasks.workunit.client.0.smithi094.stdout: "accepted_pn": 0 2021-07-26T06:09:22.287 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:22.287 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:09:22.288 INFO:tasks.workunit.client.0.smithi094.stderr:report 2817986145 2021-07-26T06:09:22.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: return 1 2021-07-26T06:09:22.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2021-07-26T06:09:22.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2021-07-26T06:09:22.289 INFO:tasks.workunit.client.0.smithi094.stdout:writing non-divergent object existing_4 2021-07-26T06:09:22.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2021-07-26T06:09:22.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:09:22.401 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:09:22.401 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2021-07-26T06:08:46.712763+0000 29'100 43:124 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:09:22.401 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:09:22.402 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:09:22.404 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:09:22.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2021-07-26T06:09:22.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:09:22.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2021-07-26T06:09:22.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:09:22.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:09:22.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:09:22.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:09:22.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:09:22.545 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:09:22.545 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:09:22.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:09:22.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:09:22.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:09:22.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:09:22.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:09:22.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:09:22.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:09:22.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:09:23.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:09:23.027 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:09:23.027 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:09:23.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:09:23.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:09:23.028 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:09:23.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=154618822670 2021-07-26T06:09:23.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 154618822670 2021-07-26T06:09:23.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-154618822670' 2021-07-26T06:09:23.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:09:23.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:09:23.281 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-07-26T06:09:23.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T06:09:23.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T06:09:23.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T06:09:23.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:09:23.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:09:23.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=163208757261 2021-07-26T06:09:23.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 163208757261 2021-07-26T06:09:23.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2021-07-26T06:09:23.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:09:23.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-154618822670 2021-07-26T06:09:23.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:09:23.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:09:23.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-154618822670 2021-07-26T06:09:23.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:09:23.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=154618822670 2021-07-26T06:09:23.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2021-07-26T06:09:23.440 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 154618822670 2021-07-26T06:09:23.441 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:09:23.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 154618822667 -lt 154618822670 2021-07-26T06:09:23.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:09:24.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:09:24.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:09:25.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 154618822670 -lt 154618822670 2021-07-26T06:09:25.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:09:25.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-163208757261 2021-07-26T06:09:25.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:09:25.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:09:25.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:09:25.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-163208757261 2021-07-26T06:09:25.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=163208757261 2021-07-26T06:09:25.210 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 163208757261 2021-07-26T06:09:25.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2021-07-26T06:09:25.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:09:25.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757261 -lt 163208757261 2021-07-26T06:09:25.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:09:25.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:25.618 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:26.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:09:26.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:26.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:26.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:26.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:26.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:26.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:26.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:26.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:26.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:26.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:26.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:26.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:26.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:09:26.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:09:26.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:09:26.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:26.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:09:27.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:27.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:27.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:27.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:27.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:27.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:27.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:27.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:27.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:27.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:27.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:27.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:27.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:27.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:27.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:27.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:27.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:27.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:27.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:27.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:27.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:28.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:28.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:28.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:09:28.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:28.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:09:28.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:28.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:28.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:28.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:28.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:28.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:28.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:28.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:28.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:28.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:28.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:28.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:29.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:29.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:29.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:29.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:29.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:29.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:29.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:29.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:29.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:29.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:29.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:29.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:09:29.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:29.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:09:30.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:30.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:30.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:30.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:30.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:30.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:30.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:30.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:30.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:30.705 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:30.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:30.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:31.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:31.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:31.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:31.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:31.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:31.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:31.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:31.166 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:31.166 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:31.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:31.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:31.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:09:31.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:31.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:09:32.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:32.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:32.451 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:32.451 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:32.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:32.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:32.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:32.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:32.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:32.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:32.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:32.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:33.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:33.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:33.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:33.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:33.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:33.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:33.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:33.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:33.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:33.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:33.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:33.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 4 >= 8 )) 2021-07-26T06:09:33.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:33.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 1.6 2021-07-26T06:09:35.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:35.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:35.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:35.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:35.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:35.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:35.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:35.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:35.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:35.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:35.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:35.733 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:36.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:36.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:36.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:36.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:36.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:36.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:36.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:36.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:36.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:36.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:36.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:36.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 5 >= 8 )) 2021-07-26T06:09:36.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:36.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 3.2 2021-07-26T06:09:39.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:39.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:39.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:39.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:39.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:39.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:39.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:39.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:40.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:40.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:40.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:40.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:40.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:40.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:40.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:40.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:40.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:40.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:40.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:40.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:40.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:41.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:41.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:41.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 6 >= 8 )) 2021-07-26T06:09:41.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:41.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 6.4 2021-07-26T06:09:47.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:47.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:47.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:47.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:47.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:47.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:47.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:47.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:47.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:47.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:47.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:47.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:48.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:48.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:48.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:48.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:48.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:48.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:48.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:48.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:48.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:48.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:48.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:48.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 7 >= 8 )) 2021-07-26T06:09:48.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:09:48.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 7.3 2021-07-26T06:09:56.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:09:56.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:09:56.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:09:56.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:09:56.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:09:56.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:09:56.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:09:56.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:09:56.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:09:56.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:09:56.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:09:56.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:09:57.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:09:57.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:09:57.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:09:57.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:09:57.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:09:57.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:09:57.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:09:57.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:09:57.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:09:57.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:09:57.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:09:57.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 8 >= 8 )) 2021-07-26T06:09:57.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph report 2021-07-26T06:09:58.034 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:09:58.034 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_fingerprint": "911f13a8-971e-4795-8d0c-76509363bcf1", 2021-07-26T06:09:58.034 INFO:tasks.workunit.client.0.smithi094.stdout: "version": "17.0.0-6358-g197e564f", 2021-07-26T06:09:58.035 INFO:tasks.workunit.client.0.smithi094.stdout: "commit": "197e564fdaee12596de1dc8f3253d1704ae66d5d", 2021-07-26T06:09:58.035 INFO:tasks.workunit.client.0.smithi094.stdout: "timestamp": "2021-07-26T06:09:58.041700+0000", 2021-07-26T06:09:58.035 INFO:tasks.workunit.client.0.smithi094.stdout: "tag": "", 2021-07-26T06:09:58.035 INFO:tasks.workunit.client.0.smithi094.stdout: "health": { 2021-07-26T06:09:58.036 INFO:tasks.workunit.client.0.smithi094.stdout: "status": "HEALTH_WARN", 2021-07-26T06:09:58.036 INFO:tasks.workunit.client.0.smithi094.stdout: "checks": { 2021-07-26T06:09:58.036 INFO:tasks.workunit.client.0.smithi094.stdout: "OSDMAP_FLAGS": { 2021-07-26T06:09:58.036 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:58.036 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:58.037 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "nodown,noout,noin flag(s) set", 2021-07-26T06:09:58.037 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 17 2021-07-26T06:09:58.037 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.037 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [], 2021-07-26T06:09:58.038 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:58.038 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.038 INFO:tasks.workunit.client.0.smithi094.stdout: "OSD_DOWN": { 2021-07-26T06:09:58.038 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:58.039 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:58.039 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 osds down", 2021-07-26T06:09:58.039 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:09:58.039 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.039 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:09:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "osd.1 (root=default,host=smithi094) is down" 2021-07-26T06:09:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:58.041 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.041 INFO:tasks.workunit.client.0.smithi094.stdout: "PG_DEGRADED": { 2021-07-26T06:09:58.041 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:58.041 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:58.042 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2021-07-26T06:09:58.042 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 2 2021-07-26T06:09:58.042 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.042 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:09:58.042 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.043 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:09:58.043 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.043 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.043 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:58.043 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.044 INFO:tasks.workunit.client.0.smithi094.stdout: "POOL_APP_NOT_ENABLED": { 2021-07-26T06:09:58.044 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:09:58.044 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:09:58.044 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 pool(s) do not have an application enabled", 2021-07-26T06:09:58.045 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:09:58.045 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.045 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:09:58.045 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.045 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "application not enabled on pool 'test'" 2021-07-26T06:09:58.046 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.046 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.046 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-07-26T06:09:58.046 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.047 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.047 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:09:58.047 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.047 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.047 INFO:tasks.workunit.client.0.smithi094.stdout: "mutes": [] 2021-07-26T06:09:58.048 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.048 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_first_committed": 1, 2021-07-26T06:09:58.048 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_last_committed": 1, 2021-07-26T06:09:58.048 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap": { 2021-07-26T06:09:58.048 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:09:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:09:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:07:08.711461Z", 2021-07-26T06:09:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:07:08.711461Z", 2021-07-26T06:09:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release": 17, 2021-07-26T06:09:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release_name": "quincy", 2021-07-26T06:09:58.050 INFO:tasks.workunit.client.0.smithi094.stdout: "election_strategy": 1, 2021-07-26T06:09:58.050 INFO:tasks.workunit.client.0.smithi094.stdout: "disallowed_leaders: ": "", 2021-07-26T06:09:58.050 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": false, 2021-07-26T06:09:58.050 INFO:tasks.workunit.client.0.smithi094.stdout: "features": { 2021-07-26T06:09:58.051 INFO:tasks.workunit.client.0.smithi094.stdout: "persistent": [ 2021-07-26T06:09:58.051 INFO:tasks.workunit.client.0.smithi094.stdout: "kraken", 2021-07-26T06:09:58.051 INFO:tasks.workunit.client.0.smithi094.stdout: "luminous", 2021-07-26T06:09:58.051 INFO:tasks.workunit.client.0.smithi094.stdout: "mimic", 2021-07-26T06:09:58.051 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap-prune", 2021-07-26T06:09:58.052 INFO:tasks.workunit.client.0.smithi094.stdout: "nautilus", 2021-07-26T06:09:58.052 INFO:tasks.workunit.client.0.smithi094.stdout: "octopus", 2021-07-26T06:09:58.052 INFO:tasks.workunit.client.0.smithi094.stdout: "pacific", 2021-07-26T06:09:58.052 INFO:tasks.workunit.client.0.smithi094.stdout: "elector-pinging", 2021-07-26T06:09:58.053 INFO:tasks.workunit.client.0.smithi094.stdout: "quincy" 2021-07-26T06:09:58.053 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.053 INFO:tasks.workunit.client.0.smithi094.stdout: "optional": [] 2021-07-26T06:09:58.053 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.053 INFO:tasks.workunit.client.0.smithi094.stdout: "mons": [ 2021-07-26T06:09:58.054 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.054 INFO:tasks.workunit.client.0.smithi094.stdout: "rank": 0, 2021-07-26T06:09:58.054 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "a", 2021-07-26T06:09:58.054 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115", 2021-07-26T06:09:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 0 2021-07-26T06:09:58.056 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.056 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.056 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.056 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115/0", 2021-07-26T06:09:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:7115/0", 2021-07-26T06:09:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 0, 2021-07-26T06:09:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 0, 2021-07-26T06:09:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_location": "{}" 2021-07-26T06:09:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: "quorum": [ 2021-07-26T06:09:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: 0 2021-07-26T06:09:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap": { 2021-07-26T06:09:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 43, 2021-07-26T06:09:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:09:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:07:09.635149+0000", 2021-07-26T06:09:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:08:46.267898+0000", 2021-07-26T06:09:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "last_up_change": "2021-07-26T06:08:43.043459+0000", 2021-07-26T06:09:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "last_in_change": "2021-07-26T06:08:45.598209+0000", 2021-07-26T06:09:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-07-26T06:09:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_num": 5800384, 2021-07-26T06:09:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_set": [ 2021-07-26T06:09:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "nodown", 2021-07-26T06:09:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "noin", 2021-07-26T06:09:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "noout", 2021-07-26T06:09:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: "pglog_hardlimit", 2021-07-26T06:09:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snapdirs", 2021-07-26T06:09:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: "recovery_deletes", 2021-07-26T06:09:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: "sortbitwise" 2021-07-26T06:09:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_version": 9, 2021-07-26T06:09:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "full_ratio": 0.99000000953674316, 2021-07-26T06:09:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-07-26T06:09:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "nearfull_ratio": 0.85000002384185791, 2021-07-26T06:09:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_snapshot": "", 2021-07-26T06:09:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_max": 1, 2021-07-26T06:09:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "max_osd": 3, 2021-07-26T06:09:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "require_min_compat_client": "luminous", 2021-07-26T06:09:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "min_compat_client": "jewel", 2021-07-26T06:09:58.065 INFO:tasks.workunit.client.0.smithi094.stdout: "require_osd_release": "quincy", 2021-07-26T06:09:58.065 INFO:tasks.workunit.client.0.smithi094.stdout: "pools": [ 2021-07-26T06:09:58.065 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.065 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:09:58.065 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_name": "test", 2021-07-26T06:09:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "create_time": "2021-07-26T06:07:48.003331+0000", 2021-07-26T06:09:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": 1, 2021-07-26T06:09:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_names": "hashpspool", 2021-07-26T06:09:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:09:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "size": 3, 2021-07-26T06:09:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "min_size": 2, 2021-07-26T06:09:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_rule": 0, 2021-07-26T06:09:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_count": 0, 2021-07-26T06:09:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_target": 0, 2021-07-26T06:09:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_barrier": 0, 2021-07-26T06:09:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-07-26T06:09:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "object_hash": 2, 2021-07-26T06:09:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_autoscale_mode": "off", 2021-07-26T06:09:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num": 1, 2021-07-26T06:09:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num": 1, 2021-07-26T06:09:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num_target": 1, 2021-07-26T06:09:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_target": 1, 2021-07-26T06:09:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_pending": 1, 2021-07-26T06:09:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pg_merge_meta": { 2021-07-26T06:09:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "source_pgid": "0.0", 2021-07-26T06:09:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "ready_epoch": 0, 2021-07-26T06:09:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_started": 0, 2021-07-26T06:09:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": 0, 2021-07-26T06:09:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "source_version": "0'0", 2021-07-26T06:09:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: "target_version": "0'0" 2021-07-26T06:09:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: "last_change": "29", 2021-07-26T06:09:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend": "0", 2021-07-26T06:09:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_prenautilus": "0", 2021-07-26T06:09:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_preluminous": "0", 2021-07-26T06:09:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "auid": 0, 2021-07-26T06:09:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_mode": "selfmanaged", 2021-07-26T06:09:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_seq": 0, 2021-07-26T06:09:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_epoch": 0, 2021-07-26T06:09:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_snaps": [], 2021-07-26T06:09:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps": "[]", 2021-07-26T06:09:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_bytes": 0, 2021-07-26T06:09:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_objects": 0, 2021-07-26T06:09:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "tiers": [], 2021-07-26T06:09:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "tier_of": -1, 2021-07-26T06:09:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "read_tier": -1, 2021-07-26T06:09:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "write_tier": -1, 2021-07-26T06:09:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_mode": "none", 2021-07-26T06:09:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_bytes": 0, 2021-07-26T06:09:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_objects": 0, 2021-07-26T06:09:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-07-26T06:09:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-07-26T06:09:58.076 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_full_ratio_micro": 800000, 2021-07-26T06:09:58.076 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_flush_age": 0, 2021-07-26T06:09:58.076 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_evict_age": 0, 2021-07-26T06:09:58.076 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profile": "", 2021-07-26T06:09:58.076 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_params": { 2021-07-26T06:09:58.077 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "none" 2021-07-26T06:09:58.077 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.077 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_period": 0, 2021-07-26T06:09:58.077 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_count": 0, 2021-07-26T06:09:58.077 INFO:tasks.workunit.client.0.smithi094.stdout: "use_gmt_hitset": true, 2021-07-26T06:09:58.078 INFO:tasks.workunit.client.0.smithi094.stdout: "min_read_recency_for_promote": 0, 2021-07-26T06:09:58.078 INFO:tasks.workunit.client.0.smithi094.stdout: "min_write_recency_for_promote": 0, 2021-07-26T06:09:58.078 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_grade_decay_rate": 0, 2021-07-26T06:09:58.078 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_search_last_n": 0, 2021-07-26T06:09:58.078 INFO:tasks.workunit.client.0.smithi094.stdout: "grade_table": [], 2021-07-26T06:09:58.079 INFO:tasks.workunit.client.0.smithi094.stdout: "stripe_width": 0, 2021-07-26T06:09:58.079 INFO:tasks.workunit.client.0.smithi094.stdout: "expected_num_objects": 0, 2021-07-26T06:09:58.079 INFO:tasks.workunit.client.0.smithi094.stdout: "fast_read": false, 2021-07-26T06:09:58.079 INFO:tasks.workunit.client.0.smithi094.stdout: "options": {}, 2021-07-26T06:09:58.079 INFO:tasks.workunit.client.0.smithi094.stdout: "application_metadata": {} 2021-07-26T06:09:58.080 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.080 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.080 INFO:tasks.workunit.client.0.smithi094.stdout: "osds": [ 2021-07-26T06:09:58.080 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.081 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:58.081 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "b02e7a44-f114-4850-90a6-03564e323691", 2021-07-26T06:09:58.081 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:09:58.081 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:09:58.081 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:09:58.082 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:09:58.082 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 5, 2021-07-26T06:09:58.082 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:09:58.082 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 36, 2021-07-26T06:09:58.082 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 41, 2021-07-26T06:09:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 30, 2021-07-26T06:09:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:09:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.084 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.084 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6802", 2021-07-26T06:09:58.084 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.084 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.085 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.085 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.085 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6803", 2021-07-26T06:09:58.085 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.085 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.086 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.086 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.086 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:09:58.086 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.086 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.087 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.087 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6804", 2021-07-26T06:09:58.087 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.087 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.087 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.088 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.088 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6805", 2021-07-26T06:09:58.088 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.088 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.089 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.089 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.089 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:09:58.089 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.089 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.090 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.090 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6808", 2021-07-26T06:09:58.090 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.090 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.090 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6809", 2021-07-26T06:09:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:09:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6806", 2021-07-26T06:09:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6807", 2021-07-26T06:09:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38132 2021-07-26T06:09:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.095 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.095 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6803/38132", 2021-07-26T06:09:58.095 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6805/38132", 2021-07-26T06:09:58.095 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6809/38132", 2021-07-26T06:09:58.095 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6807/38132", 2021-07-26T06:09:58.096 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:09:58.096 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:09:58.096 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:09:58.096 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.096 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.097 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.097 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:58.097 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "a4d2525d-0097-4f8c-b13e-ee3a5e599d6d", 2021-07-26T06:09:58.097 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 0, 2021-07-26T06:09:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:09:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:09:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:09:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 0, 2021-07-26T06:09:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 0, 2021-07-26T06:09:58.099 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 10, 2021-07-26T06:09:58.099 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 23, 2021-07-26T06:09:58.099 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 31, 2021-07-26T06:09:58.099 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:09:58.099 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:58.100 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.100 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.100 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.100 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:09:58.101 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.101 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.101 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.101 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.101 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:09:58.102 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.102 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.102 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.102 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.102 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:09:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:09:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.104 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.104 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.104 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.104 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:09:58.105 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.105 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.105 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.105 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.105 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:09:58.106 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.106 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.106 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.106 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:09:58.106 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:09:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.108 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.108 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.108 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.108 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:09:58.108 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.109 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.109 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.109 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:09:58.109 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.110 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.110 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.110 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.110 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:09:58.110 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 32635 2021-07-26T06:09:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/32635", 2021-07-26T06:09:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/32635", 2021-07-26T06:09:58.112 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/32635", 2021-07-26T06:09:58.112 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/32635", 2021-07-26T06:09:58.112 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:09:58.112 INFO:tasks.workunit.client.0.smithi094.stdout: "exists" 2021-07-26T06:09:58.112 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.113 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.113 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.113 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:58.113 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "4b82adfb-7a16-42ca-ab0d-8102aa9df430", 2021-07-26T06:09:58.114 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:09:58.114 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:09:58.114 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:09:58.114 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:09:58.114 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 15, 2021-07-26T06:09:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:09:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 38, 2021-07-26T06:09:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 0, 2021-07-26T06:09:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 32, 2021-07-26T06:09:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:09:58.116 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:09:58.116 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.116 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.116 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:09:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.118 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:09:58.118 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.118 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.118 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.119 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.119 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:09:58.119 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.119 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.119 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.120 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:09:58.120 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.120 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.120 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.120 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.121 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:09:58.121 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.121 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.121 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.122 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.122 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:09:58.122 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.122 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.122 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.123 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:09:58.123 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.123 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.123 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.123 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:09:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.125 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:09:58.125 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:09:58.125 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.125 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:09:58.126 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:09:58.126 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.126 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.126 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.126 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:09:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:09:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 38710 2021-07-26T06:09:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/38710", 2021-07-26T06:09:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/38710", 2021-07-26T06:09:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/38710", 2021-07-26T06:09:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/38710", 2021-07-26T06:09:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:09:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:09:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:09:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.130 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.130 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_xinfo": [ 2021-07-26T06:09:58.130 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.130 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:58.130 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:08:31.382366+0000", 2021-07-26T06:09:58.131 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:09:58.131 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:09:58.131 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:09:58.131 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:09:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:07:21.271225+0000", 2021-07-26T06:09:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:09:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.132 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:08:31.967926+0000", 2021-07-26T06:09:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:09:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:09:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:09:58.133 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:09:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:07:33.240355+0000", 2021-07-26T06:09:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:09:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.134 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:08:32.551830+0000", 2021-07-26T06:09:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:09:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:09:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:09:58.136 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:09:58.136 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:07:44.587495+0000", 2021-07-26T06:09:58.136 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:09:58.136 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.136 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.137 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap": [], 2021-07-26T06:09:58.137 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap_items": [], 2021-07-26T06:09:58.137 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_temp": [], 2021-07-26T06:09:58.137 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_temp": [], 2021-07-26T06:09:58.137 INFO:tasks.workunit.client.0.smithi094.stdout: "blocklist": {}, 2021-07-26T06:09:58.138 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profiles": { 2021-07-26T06:09:58.138 INFO:tasks.workunit.client.0.smithi094.stdout: "default": { 2021-07-26T06:09:58.138 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:09:58.138 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "2", 2021-07-26T06:09:58.139 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:09:58.139 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van" 2021-07-26T06:09:58.139 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.139 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.139 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps_queue": [], 2021-07-26T06:09:58.140 INFO:tasks.workunit.client.0.smithi094.stdout: "new_removed_snaps": [], 2021-07-26T06:09:58.140 INFO:tasks.workunit.client.0.smithi094.stdout: "new_purged_snaps": [], 2021-07-26T06:09:58.140 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_node_flags": {}, 2021-07-26T06:09:58.140 INFO:tasks.workunit.client.0.smithi094.stdout: "device_class_flags": {}, 2021-07-26T06:09:58.140 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": { 2021-07-26T06:09:58.141 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_enabled": false, 2021-07-26T06:09:58.141 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_bucket_count": 0, 2021-07-26T06:09:58.141 INFO:tasks.workunit.client.0.smithi094.stdout: "degraded_stretch_mode": 0, 2021-07-26T06:09:58.141 INFO:tasks.workunit.client.0.smithi094.stdout: "recovering_stretch_mode": 0, 2021-07-26T06:09:58.141 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_bucket": 0 2021-07-26T06:09:58.142 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.142 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.142 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_metadata": [ 2021-07-26T06:09:58.142 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.143 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:58.143 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:09:58.143 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6804/38132,v1:127.0.0.1:6805/38132]", 2021-07-26T06:09:58.143 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:09:58.143 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:09:58.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:09:58.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:09:58.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:09:58.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:09:58.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:09:58.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:09:58.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:09:58.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-07-26T06:09:58.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:09:58.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:09:58.146 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:09:58.146 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:09:58.146 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:09:58.146 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:09:58.147 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:09:58.147 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:09:58.147 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:09:58.147 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:09:58.147 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:09:58.148 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:09:58.148 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:09:58.148 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:09:58.148 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:09:58.148 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6802/38132,v1:127.0.0.1:6803/38132]", 2021-07-26T06:09:58.149 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:09:58.149 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/38132,v1:127.0.0.1:6809/38132]", 2021-07-26T06:09:58.149 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/38132,v1:127.0.0.1:6807/38132]", 2021-07-26T06:09:58.149 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:09:58.149 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:09:58.150 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:09:58.150 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:09:58.150 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:09:58.150 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:09:58.151 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:09:58.151 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:09:58.151 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:09:58.151 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/0", 2021-07-26T06:09:58.151 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:09:58.152 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:09:58.152 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:09:58.152 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.152 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.152 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:58.153 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:09:58.153 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/32635,v1:127.0.0.1:6813/32635]", 2021-07-26T06:09:58.153 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:09:58.153 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:09:58.153 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:09:58.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:09:58.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:09:58.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:09:58.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:09:58.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:09:58.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:09:58.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-07-26T06:09:58.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:09:58.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:09:58.156 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:09:58.156 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:09:58.156 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:09:58.156 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:09:58.156 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:09:58.157 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:09:58.157 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:09:58.157 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:09:58.157 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:09:58.157 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:09:58.158 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:09:58.158 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:09:58.158 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:09:58.158 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/32635,v1:127.0.0.1:6811/32635]", 2021-07-26T06:09:58.158 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:09:58.159 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/32635,v1:127.0.0.1:6817/32635]", 2021-07-26T06:09:58.159 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/32635,v1:127.0.0.1:6815/32635]", 2021-07-26T06:09:58.159 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:09:58.159 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:09:58.160 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:09:58.160 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:09:58.160 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:09:58.160 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:09:58.160 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:09:58.161 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:09:58.161 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:09:58.161 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/1", 2021-07-26T06:09:58.161 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:09:58.161 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:09:58.162 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:09:58.162 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.162 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.162 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:58.162 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:09:58.163 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/38710,v1:127.0.0.1:6813/38710]", 2021-07-26T06:09:58.163 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:09:58.163 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:09:58.163 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:09:58.163 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:09:58.164 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:09:58.164 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:09:58.164 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:09:58.164 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:09:58.164 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:09:58.165 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-07-26T06:09:58.165 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:09:58.165 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:09:58.165 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:09:58.165 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:09:58.166 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:09:58.166 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:09:58.166 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:09:58.166 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:09:58.167 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:09:58.167 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:09:58.167 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:09:58.167 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:09:58.167 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:09:58.168 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:09:58.168 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:09:58.168 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/38710,v1:127.0.0.1:6811/38710]", 2021-07-26T06:09:58.168 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:09:58.168 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/38710,v1:127.0.0.1:6817/38710]", 2021-07-26T06:09:58.169 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/38710,v1:127.0.0.1:6815/38710]", 2021-07-26T06:09:58.169 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:09:58.169 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:09:58.169 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:09:58.169 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:09:58.170 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:09:58.170 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:09:58.170 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:09:58.170 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:09:58.170 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:09:58.171 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/2", 2021-07-26T06:09:58.171 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:09:58.171 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:09:58.171 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:09:58.172 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.172 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.172 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_clean_epochs": { 2021-07-26T06:09:58.172 INFO:tasks.workunit.client.0.smithi094.stdout: "min_last_epoch_clean": 0, 2021-07-26T06:09:58.172 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": { 2021-07-26T06:09:58.173 INFO:tasks.workunit.client.0.smithi094.stdout: "per_pool": [] 2021-07-26T06:09:58.173 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.173 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_epochs": [ 2021-07-26T06:09:58.173 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.173 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:58.174 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 36 2021-07-26T06:09:58.174 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.174 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.174 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:58.174 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 10 2021-07-26T06:09:58.175 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.175 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.175 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:58.175 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 38 2021-07-26T06:09:58.175 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.176 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.176 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.176 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_first_committed": 1, 2021-07-26T06:09:58.176 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_last_committed": 43, 2021-07-26T06:09:58.177 INFO:tasks.workunit.client.0.smithi094.stdout: "crushmap": { 2021-07-26T06:09:58.177 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": [ 2021-07-26T06:09:58.177 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.177 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:58.177 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.0", 2021-07-26T06:09:58.178 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:09:58.178 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.178 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.178 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:58.178 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.1", 2021-07-26T06:09:58.179 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:09:58.179 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.179 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.179 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:58.180 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.2", 2021-07-26T06:09:58.180 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:09:58.180 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.180 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.180 INFO:tasks.workunit.client.0.smithi094.stdout: "types": [ 2021-07-26T06:09:58.181 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.181 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 0, 2021-07-26T06:09:58.181 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd" 2021-07-26T06:09:58.181 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.181 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.182 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:09:58.182 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "host" 2021-07-26T06:09:58.182 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.182 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.183 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 2, 2021-07-26T06:09:58.183 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "chassis" 2021-07-26T06:09:58.183 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.183 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.183 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 3, 2021-07-26T06:09:58.184 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "rack" 2021-07-26T06:09:58.184 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.184 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.184 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 4, 2021-07-26T06:09:58.184 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "row" 2021-07-26T06:09:58.185 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.185 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.185 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 5, 2021-07-26T06:09:58.185 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pdu" 2021-07-26T06:09:58.186 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.186 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.186 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 6, 2021-07-26T06:09:58.186 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pod" 2021-07-26T06:09:58.186 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.187 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.187 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 7, 2021-07-26T06:09:58.187 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "room" 2021-07-26T06:09:58.187 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.188 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.188 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 8, 2021-07-26T06:09:58.188 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "datacenter" 2021-07-26T06:09:58.188 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.188 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.189 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 9, 2021-07-26T06:09:58.189 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "zone" 2021-07-26T06:09:58.189 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.189 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.189 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 10, 2021-07-26T06:09:58.190 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "region" 2021-07-26T06:09:58.190 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.190 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.190 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:09:58.191 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "root" 2021-07-26T06:09:58.191 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.191 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.191 INFO:tasks.workunit.client.0.smithi094.stdout: "buckets": [ 2021-07-26T06:09:58.191 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.192 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -1, 2021-07-26T06:09:58.192 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default", 2021-07-26T06:09:58.192 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:09:58.192 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:09:58.193 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:58.193 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:58.193 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:58.193 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:58.193 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.194 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:09:58.194 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:58.194 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:58.194 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.194 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.195 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.195 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.195 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -2, 2021-07-26T06:09:58.195 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default~hdd", 2021-07-26T06:09:58.195 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:09:58.196 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:09:58.196 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:58.196 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:58.196 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:58.196 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:58.197 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.197 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:09:58.197 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:58.197 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:58.197 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.198 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.198 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.198 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.198 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:09:58.199 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094", 2021-07-26T06:09:58.199 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:09:58.199 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:09:58.199 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:58.199 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:58.200 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:58.200 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:58.200 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.200 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:58.200 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:58.201 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:58.201 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.201 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.201 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:58.201 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:58.202 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:09:58.202 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.202 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.202 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:58.203 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:58.203 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:09:58.203 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.203 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.203 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.204 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.204 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:09:58.204 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094~hdd", 2021-07-26T06:09:58.204 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:09:58.204 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:09:58.205 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:09:58.205 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:09:58.205 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:09:58.205 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:09:58.205 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.206 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:09:58.206 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:58.206 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:09:58.206 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.207 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.207 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:09:58.207 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:58.207 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:09:58.207 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.208 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.208 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:09:58.208 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:09:58.208 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:09:58.208 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.209 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.209 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.209 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.209 INFO:tasks.workunit.client.0.smithi094.stdout: "rules": [ 2021-07-26T06:09:58.209 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.210 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 0, 2021-07-26T06:09:58.210 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "replicated_rule", 2021-07-26T06:09:58.210 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:09:58.210 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:09:58.211 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.211 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:09:58.211 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:09:58.211 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:09:58.211 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.212 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.212 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_firstn", 2021-07-26T06:09:58.212 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:09:58.212 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:09:58.213 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.213 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.213 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:09:58.213 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.213 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:09:58.214 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.214 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.214 INFO:tasks.workunit.client.0.smithi094.stdout: "tunables": { 2021-07-26T06:09:58.214 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_tries": 0, 2021-07-26T06:09:58.214 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_fallback_tries": 0, 2021-07-26T06:09:58.215 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_total_tries": 50, 2021-07-26T06:09:58.215 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_descend_once": 1, 2021-07-26T06:09:58.215 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_vary_r": 1, 2021-07-26T06:09:58.215 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_stable": 1, 2021-07-26T06:09:58.216 INFO:tasks.workunit.client.0.smithi094.stdout: "straw_calc_version": 1, 2021-07-26T06:09:58.216 INFO:tasks.workunit.client.0.smithi094.stdout: "allowed_bucket_algs": 54, 2021-07-26T06:09:58.216 INFO:tasks.workunit.client.0.smithi094.stdout: "profile": "jewel", 2021-07-26T06:09:58.216 INFO:tasks.workunit.client.0.smithi094.stdout: "optimal_tunables": 1, 2021-07-26T06:09:58.216 INFO:tasks.workunit.client.0.smithi094.stdout: "legacy_tunables": 0, 2021-07-26T06:09:58.217 INFO:tasks.workunit.client.0.smithi094.stdout: "minimum_required_version": "jewel", 2021-07-26T06:09:58.217 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables": 1, 2021-07-26T06:09:58.217 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables2": 1, 2021-07-26T06:09:58.217 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v2_rules": 0, 2021-07-26T06:09:58.217 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables3": 1, 2021-07-26T06:09:58.218 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v3_rules": 0, 2021-07-26T06:09:58.218 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v4_buckets": 1, 2021-07-26T06:09:58.218 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables5": 1, 2021-07-26T06:09:58.218 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v5_rules": 0 2021-07-26T06:09:58.219 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.219 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_args": {} 2021-07-26T06:09:58.219 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.219 INFO:tasks.workunit.client.0.smithi094.stdout: "fsmap": { 2021-07-26T06:09:58.219 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:09:58.220 INFO:tasks.workunit.client.0.smithi094.stdout: "default_fscid": -1, 2021-07-26T06:09:58.220 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": { 2021-07-26T06:09:58.220 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": {}, 2021-07-26T06:09:58.220 INFO:tasks.workunit.client.0.smithi094.stdout: "ro_compat": {}, 2021-07-26T06:09:58.221 INFO:tasks.workunit.client.0.smithi094.stdout: "incompat": { 2021-07-26T06:09:58.221 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_1": "base v0.20", 2021-07-26T06:09:58.221 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_2": "client writeable ranges", 2021-07-26T06:09:58.221 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_3": "default file layouts on dirs", 2021-07-26T06:09:58.221 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_4": "dir inode in separate object", 2021-07-26T06:09:58.222 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_5": "mds uses versioned encoding", 2021-07-26T06:09:58.222 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_6": "dirfrag is stored in omap", 2021-07-26T06:09:58.222 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_8": "no anchor table", 2021-07-26T06:09:58.222 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_9": "file layout v2", 2021-07-26T06:09:58.222 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_10": "snaprealm v2" 2021-07-26T06:09:58.223 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.223 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.223 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_flags": { 2021-07-26T06:09:58.223 INFO:tasks.workunit.client.0.smithi094.stdout: "enable_multiple": true, 2021-07-26T06:09:58.224 INFO:tasks.workunit.client.0.smithi094.stdout: "ever_enabled_multiple": true 2021-07-26T06:09:58.224 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.224 INFO:tasks.workunit.client.0.smithi094.stdout: "standbys": [], 2021-07-26T06:09:58.224 INFO:tasks.workunit.client.0.smithi094.stdout: "filesystems": [] 2021-07-26T06:09:58.224 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.225 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_first_committed": 1, 2021-07-26T06:09:58.225 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_last_committed": 1, 2021-07-26T06:09:58.225 INFO:tasks.workunit.client.0.smithi094.stdout: "auth": { 2021-07-26T06:09:58.225 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:09:58.225 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 5, 2021-07-26T06:09:58.226 INFO:tasks.workunit.client.0.smithi094.stdout: "num_secrets": 3 2021-07-26T06:09:58.226 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.226 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:09:58.226 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_active": 1, 2021-07-26T06:09:58.227 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_unknown": 0, 2021-07-26T06:09:58.227 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osd": 3, 2021-07-26T06:09:58.227 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_sum": { 2021-07-26T06:09:58.227 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:09:58.227 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61843, 2021-07-26T06:09:58.228 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 100, 2021-07-26T06:09:58.228 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:09:58.228 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 300, 2021-07-26T06:09:58.228 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:09:58.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:09:58.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 100, 2021-07-26T06:09:58.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:09:58.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:09:58.229 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 100, 2021-07-26T06:09:58.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:09:58.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:09:58.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:09:58.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 101, 2021-07-26T06:09:58.230 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 101, 2021-07-26T06:09:58.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:09:58.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:09:58.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:09:58.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:09:58.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:09:58.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:09:58.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:09:58.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:09:58.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:09:58.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:09:58.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:09:58.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:09:58.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:09:58.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:09:58.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:09:58.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:09:58.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:09:58.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:09:58.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:09:58.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:09:58.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:09:58.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:09:58.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:09:58.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:09:58.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:09:58.236 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.236 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:09:58.236 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:09:58.237 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:09:58.237 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:58.237 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 0, 2021-07-26T06:09:58.237 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 0, 2021-07-26T06:09:58.238 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:58.238 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:58.238 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:58.238 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:09:58.238 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:09:58.239 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.239 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 17, 2021-07-26T06:09:58.239 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 17, 2021-07-26T06:09:58.239 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 2, 2021-07-26T06:09:58.240 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 2, 2021-07-26T06:09:58.240 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 0 2021-07-26T06:09:58.240 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.240 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum": { 2021-07-26T06:09:58.240 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:09:58.241 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:09:58.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:09:58.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:09:58.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:09:58.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:09:58.242 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:09:58.242 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18488, 2021-07-26T06:09:58.242 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2136, 2021-07-26T06:09:58.242 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:09:58.243 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16317, 2021-07-26T06:09:58.243 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554312, 2021-07-26T06:09:58.243 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:09:58.243 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:09:58.243 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103615488, 2021-07-26T06:09:58.244 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:58.244 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2187264, 2021-07-26T06:09:58.244 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 468880, 2021-07-26T06:09:58.244 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:58.244 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:58.245 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:58.245 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:09:58.245 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16709520 2021-07-26T06:09:58.245 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.245 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:09:58.246 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:09:58.246 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:09:58.246 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:09:58.246 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:09:58.246 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:09:58.247 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:09:58.247 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.247 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:09:58.247 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:09:58.248 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:09:58.248 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:09:58.248 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:09:58.248 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.248 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:09:58.249 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:09:58.249 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.249 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum_by_class": { 2021-07-26T06:09:58.249 INFO:tasks.workunit.client.0.smithi094.stdout: "hdd": { 2021-07-26T06:09:58.249 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:09:58.250 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:09:58.250 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:09:58.250 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:09:58.250 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:09:58.251 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:09:58.251 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:09:58.251 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18488, 2021-07-26T06:09:58.251 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2136, 2021-07-26T06:09:58.251 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:09:58.252 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16317, 2021-07-26T06:09:58.252 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554312, 2021-07-26T06:09:58.252 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:09:58.252 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:09:58.252 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103615488, 2021-07-26T06:09:58.253 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:58.253 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2187264, 2021-07-26T06:09:58.253 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 468880, 2021-07-26T06:09:58.253 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:58.253 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:58.254 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:58.254 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:09:58.254 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16709520 2021-07-26T06:09:58.254 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.254 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:09:58.255 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:09:58.255 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:09:58.255 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:09:58.255 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:09:58.256 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:09:58.256 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:09:58.256 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.256 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:09:58.256 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:09:58.257 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:09:58.257 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:09:58.257 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:09:58.257 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.257 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:09:58.258 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:09:58.258 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.258 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.258 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_stats": [ 2021-07-26T06:09:58.258 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.259 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:09:58.259 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:09:58.259 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:09:58.259 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61843, 2021-07-26T06:09:58.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 100, 2021-07-26T06:09:58.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:09:58.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 300, 2021-07-26T06:09:58.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:09:58.260 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:09:58.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 100, 2021-07-26T06:09:58.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:09:58.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:09:58.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 100, 2021-07-26T06:09:58.261 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:09:58.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:09:58.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:09:58.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 101, 2021-07-26T06:09:58.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 101, 2021-07-26T06:09:58.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:09:58.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:09:58.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:09:58.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:09:58.263 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:09:58.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:09:58.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:09:58.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:09:58.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:09:58.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:09:58.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:09:58.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:09:58.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:09:58.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:09:58.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:09:58.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:09:58.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:09:58.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:09:58.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:09:58.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:09:58.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:09:58.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:09:58.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:09:58.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:09:58.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:09:58.268 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.268 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:09:58.268 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:09:58.269 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:09:58.269 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:09:58.269 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 1220608, 2021-07-26T06:09:58.269 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 184438, 2021-07-26T06:09:58.270 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:09:58.270 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:09:58.270 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:09:58.270 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:09:58.270 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:09:58.271 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.271 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 17, 2021-07-26T06:09:58.271 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 17, 2021-07-26T06:09:58.271 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 2, 2021-07-26T06:09:58.271 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 2, 2021-07-26T06:09:58.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 3 2021-07-26T06:09:58.272 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.272 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.272 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_stats": [ 2021-07-26T06:09:58.272 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.273 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:58.273 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 154618822676 2021-07-26T06:09:58.273 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.273 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.274 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:58.274 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 42949672978 2021-07-26T06:09:58.274 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.274 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.274 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:58.275 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 163208757267 2021-07-26T06:09:58.275 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.275 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.275 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_state": [ 2021-07-26T06:09:58.275 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.276 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized+degraded", 2021-07-26T06:09:58.276 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 1 2021-07-26T06:09:58.276 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.276 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.277 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_osd": [ 2021-07-26T06:09:58.277 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.277 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:09:58.277 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:09:58.277 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:09:58.278 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:09:58.278 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.278 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.278 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:09:58.278 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 1, 2021-07-26T06:09:58.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:09:58.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:09:58.279 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.279 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.279 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:09:58.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:09:58.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 0, 2021-07-26T06:09:58.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:09:58.280 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.281 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.281 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": [ 2021-07-26T06:09:58.281 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:09:58.281 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:09:58.281 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": {} 2021-07-26T06:09:58.282 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.282 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:09:58.282 INFO:tasks.workunit.client.0.smithi094.stdout: "servicemap": { 2021-07-26T06:09:58.282 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 6, 2021-07-26T06:09:58.282 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:09:44.383367+0000", 2021-07-26T06:09:58.283 INFO:tasks.workunit.client.0.smithi094.stdout: "services": {} 2021-07-26T06:09:58.283 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:09:58.283 INFO:tasks.workunit.client.0.smithi094.stdout: "paxos": { 2021-07-26T06:09:58.283 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:09:58.284 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 475, 2021-07-26T06:09:58.284 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pn": 0, 2021-07-26T06:09:58.284 INFO:tasks.workunit.client.0.smithi094.stdout: "accepted_pn": 0 2021-07-26T06:09:58.284 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:09:58.284 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:09:58.285 INFO:tasks.workunit.client.0.smithi094.stderr:report 3249593356 2021-07-26T06:09:58.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: return 1 2021-07-26T06:09:58.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2021-07-26T06:09:58.286 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery 2021-07-26T06:09:58.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:09:58.424 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:09:58.424 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:08:46.712763+0000 43'101 43:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:09:58.424 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:09:58.425 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:09:58.427 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:09:58.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:09:58.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2021-07-26T06:09:58.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:09:58.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:09:58.437 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:09:58.438 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:09:58.438 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:09:58.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:09:58.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-07-26T06:09:58.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok set_recovery_delay 100000 2021-07-26T06:09:58.527 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:09:58.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-07-26T06:09:58.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2021-07-26T06:09:58.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:09:58.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:09:58.537 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:09:58.537 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:09:58.537 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:09:58.538 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:09:58.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-07-26T06:09:58.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok set_recovery_delay 100000 2021-07-26T06:09:58.620 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:09:58.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2021-07-26T06:09:58.627 INFO:tasks.workunit.client.0.smithi094.stdout:revive divergent 1 2021-07-26T06:09:58.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:09:58.962 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:09:58.962 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:08:46.712763+0000 43'101 43:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:09:58.962 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:09:58.963 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:09:58.965 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:09:58.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2021-07-26T06:09:59.759 INFO:tasks.workunit.client.0.smithi094.stderr:noup is set 2021-07-26T06:09:59.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-07-26T06:09:59.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:09:59.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:09:59.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:09:59.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:09:59.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:09:59.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:09:59.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:09:59.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:09:59.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:09:59.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:09:59.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:09:59.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:09:59.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:09:59.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:09:59.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:09:59.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:09:59.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:09:59.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:09:59.778 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:09:59.778 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:09:59.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:09:59.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:09:59.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:09:59.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:09:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:09:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:09:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:09:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:09:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:09:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:09:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:09:59.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:09:59.782 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:09:59.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:09:59.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:09:59.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:09:59.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:09:59.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:09:59.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:09:59.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:09:59.812 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:09:59.818+0000 7fd16e31ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:09:59.816 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:09:59.822+0000 7fd16e31ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:09:59.821 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:09:59.822+0000 7fd16e31ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:00.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: activate_osd: return 0 2021-07-26T06:10:00.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2021-07-26T06:10:00.358 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:00.361+0000 7fd16e31ff40 -1 Falling back to public interface 2021-07-26T06:10:03.574 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:03.576+0000 7fd16e31ff40 -1 osd.1 29 log_to_monitors {default=true} 2021-07-26T06:10:05.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2021-07-26T06:10:05.135 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery divergent 2021-07-26T06:10:05.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:10:05.491 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:10:05.492 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:08:46.712763+0000 43'101 43:126 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:10:05.492 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:10:05.493 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:10:05.494 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:10:05.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2021-07-26T06:10:05.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:10:05.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:10:05.505 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:10:05.505 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:05.505 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:05.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:10:05.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2021-07-26T06:10:05.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.1.asok set_recovery_delay 100000 2021-07-26T06:10:05.587 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:10:05.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2021-07-26T06:10:06.300 INFO:tasks.workunit.client.0.smithi094.stderr:noup is unset 2021-07-26T06:10:06.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2021-07-26T06:10:06.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:10:06.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:10:06.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:10:06.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:10:06.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:06.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:10:06.313 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:10:06.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:06.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:10:06.812 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/38132,v1:127.0.0.1:6803/38132] [v2:127.0.0.1:6804/38132,v1:127.0.0.1:6805/38132] exists,up b02e7a44-f114-4850-90a6-03564e323691 2021-07-26T06:10:06.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:10:06.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:10:06.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:10:06.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2021-07-26T06:10:06.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:10:06.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:10:06.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:10:06.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:10:06.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:06.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:10:06.816 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:10:06.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:06.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:10:07.217 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/41486,v1:127.0.0.1:6819/41486] [v2:127.0.0.1:6820/41486,v1:127.0.0.1:6821/41486] exists,up a4d2525d-0097-4f8c-b13e-ee3a5e599d6d 2021-07-26T06:10:07.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:10:07.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:10:07.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:10:07.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2021-07-26T06:10:07.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:10:07.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:10:07.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:10:07.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:10:07.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:07.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:10:07.222 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:10:07.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:07.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:10:07.586 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/38710,v1:127.0.0.1:6811/38710] [v2:127.0.0.1:6812/38710,v1:127.0.0.1:6813/38710] exists,up 4b82adfb-7a16-42ca-ab0d-8102aa9df430 2021-07-26T06:10:07.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:10:07.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:10:07.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:10:07.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:10:07.978 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:10:07.979 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:08:46.712763+0000 43'101 43:126 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:10:07.979 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:10:07.979 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:10:07.984 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:10:07.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2021-07-26T06:10:07.997 INFO:tasks.workunit.client.0.smithi094.stdout:wait for peering 2021-07-26T06:10:07.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:10:08.324 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:10:08.324 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61843 0 0 17 17 active+undersized+degraded 2021-07-26T06:08:46.712763+0000 43'101 43:126 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:10:08.325 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:10:08.325 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:10:08.327 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:10:08.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2021-07-26T06:10:14.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2021-07-26T06:10:14.808 INFO:tasks.workunit.client.0.smithi094.stdout:killing divergent 1 2021-07-26T06:10:14.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:10:15.164 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:10:15.164 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 4 4 0 0 61843 0 0 11 11 active+recovering+degraded+wait 2021-07-26T06:10:07.174088+0000 43'101 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:10:15.165 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:10:15.165 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:10:15.167 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:10:15.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2021-07-26T06:10:15.182 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:10:15.182 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:10:15.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:10:15.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:10:15.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:10:15.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:10:15.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.18330.out 2021-07-26T06:10:15.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.18330.out 2021-07-26T06:10:15.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-07-26T06:10:15.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:10:15.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:10:15.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:10:15.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-07-26T06:10:15.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.18330.out 2021-07-26T06:10:16.358 INFO:tasks.workunit.client.0.smithi094.stderr:Exporting 1.0 info 1.0( v 49'102 lc 0'0 (29'90,49'102] local-lis/les=48/49 n=101 ec=23/23 lis/c=48/23 les/c/f=49/24/0 sis=48 pruub=7.545355320s) 2021-07-26T06:10:16.368 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:023dd3d2:::existing_12:head# 2021-07-26T06:10:16.400 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:04c40a0c:::existing_93:head# 2021-07-26T06:10:16.444 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:064ff5fb:::existing_36:head# 2021-07-26T06:10:16.465 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:091f9f1d:::existing_94:head# 2021-07-26T06:10:16.483 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:0a865a93:::existing_54:head# 2021-07-26T06:10:16.491 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:0fed513c:::existing_41:head# 2021-07-26T06:10:16.510 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:13b9e0b8:::existing_92:head# 2021-07-26T06:10:16.516 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:169157fa:::existing_81:head# 2021-07-26T06:10:16.533 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:17731382:::existing_58:head# 2021-07-26T06:10:16.552 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:18a45a39:::existing_43:head# 2021-07-26T06:10:16.558 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:1c160f01:::existing_66:head# 2021-07-26T06:10:16.566 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:1dd6c249:::existing_97:head# 2021-07-26T06:10:16.584 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:227e0c77:::existing_7:head# 2021-07-26T06:10:16.601 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:23e1b063:::existing_59:head# 2021-07-26T06:10:16.609 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:28566edb:::existing_78:head# 2021-07-26T06:10:16.625 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:2a101182:::existing_20:head# 2021-07-26T06:10:16.633 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:2d1519da:::existing_96:head# 2021-07-26T06:10:16.642 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:3078b87a:::existing_86:head# 2021-07-26T06:10:16.650 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:30bde01b:::existing_15:head# 2021-07-26T06:10:16.651 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:317a7e19:::existing_56:head# 2021-07-26T06:10:16.667 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:38b9c7fa:::existing_33:head# 2021-07-26T06:10:16.675 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:3e16d84b:::existing_48:head# 2021-07-26T06:10:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:45fe8203:::existing_75:head# 2021-07-26T06:10:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:4e82c9ba:::existing_50:head# 2021-07-26T06:10:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:5298dcdb:::existing_57:head# 2021-07-26T06:10:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:52ebaa44:::existing_32:head# 2021-07-26T06:10:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:53588912:::existing_9:head# 2021-07-26T06:10:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:5b7e2de6:::existing_84:head# 2021-07-26T06:10:16.701 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:5da0255c:::existing_13:head# 2021-07-26T06:10:16.702 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:5da2dab8:::existing_53:head# 2021-07-26T06:10:16.703 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:5ee3514b:::existing_30:head# 2021-07-26T06:10:16.704 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:602f83fe:::foo:head# 2021-07-26T06:10:16.705 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:60426cc7:::existing_62:head# 2021-07-26T06:10:16.706 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:628918ca:::existing_79:head# 2021-07-26T06:10:16.707 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:63f8ab2d:::existing_77:head# 2021-07-26T06:10:16.708 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:67e1780f:::existing_27:head# 2021-07-26T06:10:16.709 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:68143ca1:::existing_71:head# 2021-07-26T06:10:16.710 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:6bac9509:::existing_29:head# 2021-07-26T06:10:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:6bbec798:::existing_45:head# 2021-07-26T06:10:16.711 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:6cb23a40:::existing_65:head# 2021-07-26T06:10:16.712 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:7055ec5e:::existing_83:head# 2021-07-26T06:10:16.713 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:70a54e2b:::existing_87:head# 2021-07-26T06:10:16.713 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:750ace21:::existing_19:head# 2021-07-26T06:10:16.714 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:78623b1d:::existing_6:head# 2021-07-26T06:10:16.715 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:79881152:::existing_63:head# 2021-07-26T06:10:16.716 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:7a80c0d5:::existing_99:head# 2021-07-26T06:10:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:81490677:::existing_76:head# 2021-07-26T06:10:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:85ec8e5d:::existing_85:head# 2021-07-26T06:10:16.718 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:8e2abbf0:::existing_49:head# 2021-07-26T06:10:16.719 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:8ee0fe0f:::existing_52:head# 2021-07-26T06:10:16.720 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:923c5d4d:::existing_5:head# 2021-07-26T06:10:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:92731e71:::existing_89:head# 2021-07-26T06:10:16.721 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:928205cf:::existing_82:head# 2021-07-26T06:10:16.722 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:93be2267:::existing_24:head# 2021-07-26T06:10:16.723 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:99ceda2c:::existing_44:head# 2021-07-26T06:10:16.724 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:9dd917ee:::existing_26:head# 2021-07-26T06:10:16.725 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:a28805a6:::existing_72:head# 2021-07-26T06:10:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:a294a8f6:::existing_16:head# 2021-07-26T06:10:16.726 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:a2f5473a:::existing_42:head# 2021-07-26T06:10:16.727 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:a504a533:::existing_90:head# 2021-07-26T06:10:16.728 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:a691337e:::existing_61:head# 2021-07-26T06:10:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:a6e8d7ac:::existing_28:head# 2021-07-26T06:10:16.729 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:a9fde86e:::existing_18:head# 2021-07-26T06:10:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:aa309faa:::existing_68:head# 2021-07-26T06:10:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:ae40b0ed:::existing_31:head# 2021-07-26T06:10:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:b279b1fe:::existing_10:head# 2021-07-26T06:10:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:b343e277:::existing_88:head# 2021-07-26T06:10:16.733 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:b7130e90:::existing_80:head# 2021-07-26T06:10:16.734 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:bc6f7309:::existing_70:head# 2021-07-26T06:10:16.735 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:be8489b7:::existing_38:head# 2021-07-26T06:10:16.736 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:beb4e5e3:::existing_47:head# 2021-07-26T06:10:16.737 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:bebec877:::existing_74:head# 2021-07-26T06:10:16.737 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:c04347d8:::existing_60:head# 2021-07-26T06:10:16.738 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:c156033e:::existing_100:head# 2021-07-26T06:10:16.739 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:c19f5a8e:::existing_8:head# 2021-07-26T06:10:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:c3e6ebe4:::existing_17:head# 2021-07-26T06:10:16.740 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:c6c5a7c2:::existing_39:head# 2021-07-26T06:10:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:cc1d0abf:::existing_95:head# 2021-07-26T06:10:16.742 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:ccb3865b:::existing_73:head# 2021-07-26T06:10:16.743 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:cd84e19e:::existing_40:head# 2021-07-26T06:10:16.744 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:d2cd64ce:::existing_37:head# 2021-07-26T06:10:16.745 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:d41a25c9:::existing_22:head# 2021-07-26T06:10:16.746 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:d7f088da:::existing_91:head# 2021-07-26T06:10:16.746 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:d9cbd1d6:::existing_64:head# 2021-07-26T06:10:16.747 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:dcc4ff82:::existing_98:head# 2021-07-26T06:10:16.748 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:dd677bd3:::existing_35:head# 2021-07-26T06:10:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:dd6d90e7:::existing_46:head# 2021-07-26T06:10:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:e0e22077:::existing_21:head# 2021-07-26T06:10:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:e255b710:::existing_55:head# 2021-07-26T06:10:16.751 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:eff1d5db:::existing_14:head# 2021-07-26T06:10:16.752 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:f1b156bf:::existing_23:head# 2021-07-26T06:10:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:f5ce4199:::existing_34:head# 2021-07-26T06:10:16.753 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:f6809456:::existing_11:head# 2021-07-26T06:10:16.754 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:f699a2a9:::existing_67:head# 2021-07-26T06:10:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:f9baac43:::existing_25:head# 2021-07-26T06:10:16.755 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:fb76688e:::existing_51:head# 2021-07-26T06:10:16.756 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:fbb92eec:::existing_69:head# 2021-07-26T06:10:16.757 INFO:tasks.workunit.client.0.smithi094.stderr:Export successful 2021-07-26T06:10:16.758 INFO:tasks.workunit.client.0.smithi094.stdout: marking collection for removal 2021-07-26T06:10:16.758 INFO:tasks.workunit.client.0.smithi094.stdout:setting '_remove' omap key 2021-07-26T06:10:16.758 INFO:tasks.workunit.client.0.smithi094.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-07-26T06:10:16.984 INFO:tasks.workunit.client.0.smithi094.stderr:Remove successful 2021-07-26T06:10:17.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.18330.out 2021-07-26T06:10:17.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-07-26T06:10:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:10:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:10:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:10:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-07-26T06:10:17.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.18330.out 2021-07-26T06:10:18.321 INFO:tasks.workunit.client.0.smithi094.stdout:get_pg_num_history pg_num_history pg_num_history(e49 pg_nums {1={23=1}} deleted_pools ) 2021-07-26T06:10:18.322 INFO:tasks.workunit.client.0.smithi094.stdout:Importing pgid 1.0 2021-07-26T06:10:18.322 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:023dd3d2:::existing_12:head# 2021-07-26T06:10:18.322 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.396 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:04c40a0c:::existing_93:head# 2021-07-26T06:10:18.397 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.446 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:064ff5fb:::existing_36:head# 2021-07-26T06:10:18.447 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.573 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:091f9f1d:::existing_94:head# 2021-07-26T06:10:18.573 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.614 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:0a865a93:::existing_54:head# 2021-07-26T06:10:18.614 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.672 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:0fed513c:::existing_41:head# 2021-07-26T06:10:18.673 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.731 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:13b9e0b8:::existing_92:head# 2021-07-26T06:10:18.732 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.765 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:169157fa:::existing_81:head# 2021-07-26T06:10:18.765 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.840 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:17731382:::existing_58:head# 2021-07-26T06:10:18.840 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.873 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:18a45a39:::existing_43:head# 2021-07-26T06:10:18.874 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.907 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:1c160f01:::existing_66:head# 2021-07-26T06:10:18.907 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.940 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:1dd6c249:::existing_97:head# 2021-07-26T06:10:18.941 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:18.974 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:227e0c77:::existing_7:head# 2021-07-26T06:10:18.974 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.007 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:23e1b063:::existing_59:head# 2021-07-26T06:10:19.008 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.041 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:28566edb:::existing_78:head# 2021-07-26T06:10:19.041 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.110 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:2a101182:::existing_20:head# 2021-07-26T06:10:19.111 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.149 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:2d1519da:::existing_96:head# 2021-07-26T06:10:19.150 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.183 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:3078b87a:::existing_86:head# 2021-07-26T06:10:19.184 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.216 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:30bde01b:::existing_15:head# 2021-07-26T06:10:19.217 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.250 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:317a7e19:::existing_56:head# 2021-07-26T06:10:19.250 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.283 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:38b9c7fa:::existing_33:head# 2021-07-26T06:10:19.284 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.317 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:3e16d84b:::existing_48:head# 2021-07-26T06:10:19.317 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.350 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:45fe8203:::existing_75:head# 2021-07-26T06:10:19.350 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.384 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:4e82c9ba:::existing_50:head# 2021-07-26T06:10:19.384 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.450 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:5298dcdb:::existing_57:head# 2021-07-26T06:10:19.451 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.484 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:52ebaa44:::existing_32:head# 2021-07-26T06:10:19.484 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.517 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:53588912:::existing_9:head# 2021-07-26T06:10:19.518 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.551 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:5b7e2de6:::existing_84:head# 2021-07-26T06:10:19.551 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.584 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:5da0255c:::existing_13:head# 2021-07-26T06:10:19.585 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.618 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:5da2dab8:::existing_53:head# 2021-07-26T06:10:19.618 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.651 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:5ee3514b:::existing_30:head# 2021-07-26T06:10:19.652 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.707 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:602f83fe:::foo:head# 2021-07-26T06:10:19.708 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.743 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:60426cc7:::existing_62:head# 2021-07-26T06:10:19.744 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.777 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:628918ca:::existing_79:head# 2021-07-26T06:10:19.777 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.810 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:63f8ab2d:::existing_77:head# 2021-07-26T06:10:19.810 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.844 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:67e1780f:::existing_27:head# 2021-07-26T06:10:19.844 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.877 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:68143ca1:::existing_71:head# 2021-07-26T06:10:19.877 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.919 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:6bac9509:::existing_29:head# 2021-07-26T06:10:19.919 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.952 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:6bbec798:::existing_45:head# 2021-07-26T06:10:19.953 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:19.986 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:6cb23a40:::existing_65:head# 2021-07-26T06:10:19.986 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.052 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:7055ec5e:::existing_83:head# 2021-07-26T06:10:20.053 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.086 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:70a54e2b:::existing_87:head# 2021-07-26T06:10:20.086 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.119 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:750ace21:::existing_19:head# 2021-07-26T06:10:20.120 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.153 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:78623b1d:::existing_6:head# 2021-07-26T06:10:20.153 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.203 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:79881152:::existing_63:head# 2021-07-26T06:10:20.203 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.253 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:7a80c0d5:::existing_99:head# 2021-07-26T06:10:20.254 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.330 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:81490677:::existing_76:head# 2021-07-26T06:10:20.330 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.387 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:85ec8e5d:::existing_85:head# 2021-07-26T06:10:20.388 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.438 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:8e2abbf0:::existing_49:head# 2021-07-26T06:10:20.438 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.531 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:8ee0fe0f:::existing_52:head# 2021-07-26T06:10:20.531 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.655 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:923c5d4d:::existing_5:head# 2021-07-26T06:10:20.656 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.740 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:92731e71:::existing_89:head# 2021-07-26T06:10:20.740 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.832 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:928205cf:::existing_82:head# 2021-07-26T06:10:20.832 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.890 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:93be2267:::existing_24:head# 2021-07-26T06:10:20.890 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:20.961 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:99ceda2c:::existing_44:head# 2021-07-26T06:10:20.961 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.008 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:9dd917ee:::existing_26:head# 2021-07-26T06:10:21.008 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.041 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:a28805a6:::existing_72:head# 2021-07-26T06:10:21.042 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.075 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:a294a8f6:::existing_16:head# 2021-07-26T06:10:21.075 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.100 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:a2f5473a:::existing_42:head# 2021-07-26T06:10:21.100 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.150 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:a504a533:::existing_90:head# 2021-07-26T06:10:21.151 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.200 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:a691337e:::existing_61:head# 2021-07-26T06:10:21.201 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.280 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:a6e8d7ac:::existing_28:head# 2021-07-26T06:10:21.281 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.342 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:a9fde86e:::existing_18:head# 2021-07-26T06:10:21.343 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.392 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:aa309faa:::existing_68:head# 2021-07-26T06:10:21.393 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.444 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:ae40b0ed:::existing_31:head# 2021-07-26T06:10:21.444 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.485 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:b279b1fe:::existing_10:head# 2021-07-26T06:10:21.485 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.527 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:b343e277:::existing_88:head# 2021-07-26T06:10:21.527 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.591 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:b7130e90:::existing_80:head# 2021-07-26T06:10:21.591 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.636 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:bc6f7309:::existing_70:head# 2021-07-26T06:10:21.636 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.694 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:be8489b7:::existing_38:head# 2021-07-26T06:10:21.695 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.745 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:beb4e5e3:::existing_47:head# 2021-07-26T06:10:21.745 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.795 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:bebec877:::existing_74:head# 2021-07-26T06:10:21.795 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.845 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:c04347d8:::existing_60:head# 2021-07-26T06:10:21.845 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.937 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:c156033e:::existing_100:head# 2021-07-26T06:10:21.937 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:21.987 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:c19f5a8e:::existing_8:head# 2021-07-26T06:10:21.988 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.037 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:c3e6ebe4:::existing_17:head# 2021-07-26T06:10:22.038 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.088 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:c6c5a7c2:::existing_39:head# 2021-07-26T06:10:22.088 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.147 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:cc1d0abf:::existing_95:head# 2021-07-26T06:10:22.147 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.230 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:ccb3865b:::existing_73:head# 2021-07-26T06:10:22.232 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.272 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:cd84e19e:::existing_40:head# 2021-07-26T06:10:22.272 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.314 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:d2cd64ce:::existing_37:head# 2021-07-26T06:10:22.315 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.356 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:d41a25c9:::existing_22:head# 2021-07-26T06:10:22.356 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.398 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:d7f088da:::existing_91:head# 2021-07-26T06:10:22.398 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.423 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:d9cbd1d6:::existing_64:head# 2021-07-26T06:10:22.423 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.499 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:dcc4ff82:::existing_98:head# 2021-07-26T06:10:22.499 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.624 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:dd677bd3:::existing_35:head# 2021-07-26T06:10:22.624 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.875 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:dd6d90e7:::existing_46:head# 2021-07-26T06:10:22.875 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.942 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:e0e22077:::existing_21:head# 2021-07-26T06:10:22.942 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:22.992 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:e255b710:::existing_55:head# 2021-07-26T06:10:22.993 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.043 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:eff1d5db:::existing_14:head# 2021-07-26T06:10:23.043 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.093 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:f1b156bf:::existing_23:head# 2021-07-26T06:10:23.093 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.177 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:f5ce4199:::existing_34:head# 2021-07-26T06:10:23.178 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.227 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:f6809456:::existing_11:head# 2021-07-26T06:10:23.227 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.277 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:f699a2a9:::existing_67:head# 2021-07-26T06:10:23.277 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.319 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:f9baac43:::existing_25:head# 2021-07-26T06:10:23.319 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.378 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:fb76688e:::existing_51:head# 2021-07-26T06:10:23.378 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.420 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:fbb92eec:::existing_69:head# 2021-07-26T06:10:23.420 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:10:23.453 INFO:tasks.workunit.client.0.smithi094.stdout:write_pg epoch 49 info 1.0( v 49'102 lc 0'0 (29'90,49'102] local-lis/les=48/49 n=101 ec=23/23 lis/c=48/23 les/c/f=49/24/0 sis=48 pruub=7.545355320s) 2021-07-26T06:10:23.528 INFO:tasks.workunit.client.0.smithi094.stdout:Import successful 2021-07-26T06:10:24.076 INFO:tasks.workunit.client.0.smithi094.stdout:reviving divergent 1 2021-07-26T06:10:24.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2021-07-26T06:10:24.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:10:24.428 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:10:24.428 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 4 4 0 0 61843 0 0 11 11 active+recovering+degraded+wait 2021-07-26T06:10:07.174088+0000 43'101 49:126 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:10:24.428 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:10:24.429 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:10:24.431 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:10:24.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-07-26T06:10:24.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:10:24.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:10:24.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:10:24.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:10:24.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:10:24.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:10:24.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:10:24.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:10:24.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:10:24.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:10:24.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:10:24.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:10:24.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:10:24.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:10:24.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:10:24.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:10:24.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:10:24.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:10:24.445 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:10:24.445 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:24.446 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:24.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:10:24.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:10:24.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:10:24.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:10:24.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:10:24.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:10:24.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:10:24.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:10:24.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:10:24.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:10:24.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:10:24.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:10:24.450 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:10:24.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:10:24.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:10:24.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:10:24.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:10:24.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:10:24.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:10:24.475 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:24.472+0000 7fcec2d9ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:24.477 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:24.476+0000 7fcec2d9ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:24.479 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:24.476+0000 7fcec2d9ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:24.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:10:24.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:10:24.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:10:24.870 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:10:24.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:10:24.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:10:24.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:24.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:10:24.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:24.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:10:25.031 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:25.032+0000 7fcec2d9ef40 -1 Falling back to public interface 2021-07-26T06:10:25.239 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/41486,v1:127.0.0.1:6819/41486] [v2:127.0.0.1:6820/41486,v1:127.0.0.1:6821/41486] exists,up a4d2525d-0097-4f8c-b13e-ee3a5e599d6d 2021-07-26T06:10:25.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:10:25.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:10:25.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:10:25.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2021-07-26T06:10:25.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:10:25.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:10:25.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:10:25.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:10:25.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:25.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:10:25.243 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:10:25.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:25.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:10:25.618 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/41486,v1:127.0.0.1:6819/41486] [v2:127.0.0.1:6820/41486,v1:127.0.0.1:6821/41486] exists,up a4d2525d-0097-4f8c-b13e-ee3a5e599d6d 2021-07-26T06:10:25.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:10:25.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:10:25.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:10:25.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2021-07-26T06:10:28.004 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:28.003+0000 7fcec2d9ef40 -1 osd.1 49 log_to_monitors {default=true} 2021-07-26T06:10:45.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2021-07-26T06:10:45.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:10:45.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:10:45.625 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:10:45.625 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:45.626 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:45.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:10:45.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2021-07-26T06:10:45.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.1.asok dump_ops_in_flight 2021-07-26T06:10:45.718 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:10:45.719 INFO:tasks.workunit.client.0.smithi094.stdout: "ops": [], 2021-07-26T06:10:45.719 INFO:tasks.workunit.client.0.smithi094.stdout: "num_ops": 0 2021-07-26T06:10:45.719 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:10:45.726 INFO:tasks.workunit.client.0.smithi094.stdout:allowing recovery 2021-07-26T06:10:45.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2021-07-26T06:10:45.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:10:46.075 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:10:46.075 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 0 0 0 62460 0 0 12 12 active+clean 2021-07-26T06:10:30.823775+0000 49'102 54:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:10:46.075 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:10:46.076 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:10:46.077 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:10:46.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-07-26T06:10:46.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2021-07-26T06:10:46.222 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:10:46.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-07-26T06:10:46.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2021-07-26T06:10:46.374 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:10:46.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-07-26T06:10:46.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2021-07-26T06:10:46.548 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:10:46.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2021-07-26T06:10:46.561 INFO:tasks.workunit.client.0.smithi094.stdout:reading divergent objects 2021-07-26T06:10:46.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2021-07-26T06:10:46.909 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:10:46.910 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 0 0 0 62460 0 0 12 12 active+clean 2021-07-26T06:10:30.823775+0000 49'102 54:150 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:07:48.045606+0000 0'0 2021-07-26T06:07:48.045606+0000 0 2021-07-26T06:10:46.910 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:10:46.910 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:10:46.912 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:10:46.921 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2021-07-26T06:10:46.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2021-07-26T06:10:46.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:10:46.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2021-07-26T06:10:46.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:10:46.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2021-07-26T06:10:47.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:10:47.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2021-07-26T06:10:47.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:10:47.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2021-07-26T06:10:47.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2021-07-26T06:10:47.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:10:47.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2021-07-26T06:10:47.124 INFO:tasks.workunit.client.0.smithi094.stderr:error getting test/newobject_1: (2) No such file or directory 2021-07-26T06:10:47.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:10:47.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2021-07-26T06:10:47.166 INFO:tasks.workunit.client.0.smithi094.stderr:error getting test/newobject_2: (2) No such file or directory 2021-07-26T06:10:47.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2021-07-26T06:10:47.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-07-26T06:10:47.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:10:47.176 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4550.0:1 2021-07-26T06:08:20.729758+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:10:47.176 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4550.0:1 2021-07-26T06:08:20.729758+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:10:47.177 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:10:47.177 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2021-07-26T06:10:47.177 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: more recent entry found: 43'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4670.0:1 2021-07-26T06:09:22.466952+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0, already merged 2021-07-26T06:10:47.177 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4540.0:1 2021-07-26T06:08:20.620582+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.178 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4540.0:1 2021-07-26T06:08:20.620582+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.178 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:10:47.178 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2021-07-26T06:10:47.178 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-07-26T06:10:47.179 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-07-26T06:10:47.179 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4551.0:1 2021-07-26T06:08:20.732109+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:10:47.179 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4551.0:1 2021-07-26T06:08:20.732109+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:10:47.179 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-07-26T06:10:47.180 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2021-07-26T06:10:47.180 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-07-26T06:10:47.180 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:10:47.180 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2021-07-26T06:10:47.180 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4551.0:1 2021-07-26T06:08:20.732109+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:10:47.181 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-07-26T06:10:47.181 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2021-07-26T06:10:47.181 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4541.0:1 2021-07-26T06:08:20.621801+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.181 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4541.0:1 2021-07-26T06:08:20.621801+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.182 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:10:47.182 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2021-07-26T06:10:47.182 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-07-26T06:10:47.182 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:10:47.183 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2021-07-26T06:10:47.183 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4541.0:1 2021-07-26T06:08:20.621801+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.183 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-07-26T06:10:47.183 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2021-07-26T06:10:47.184 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4553.0:1 2021-07-26T06:08:20.734147+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.184 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: keeping 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4553.0:1 2021-07-26T06:08:20.734147+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.184 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:10:47.185 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'106 last_divergent_update: 29'106 2021-07-26T06:10:47.185 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-07-26T06:10:47.185 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:10:47.185 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2021-07-26T06:10:47.185 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4553.0:1 2021-07-26T06:08:20.734147+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.186 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-07-26T06:10:47.186 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2021-07-26T06:10:47.186 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4548.0:1 2021-07-26T06:08:20.726463+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.186 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4548.0:1 2021-07-26T06:08:20.726463+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:10:47.187 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:10:47.187 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2021-07-26T06:10:47.187 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-07-26T06:10:47.187 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:10:07.047+0000 7fd154839700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-07-26T06:10:47.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-07-26T06:10:47.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:10:47.189 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-07-26T06:10:47.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:10:47.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2021-07-26T06:10:47.196 INFO:tasks.workunit.client.0.smithi094.stdout:success 2021-07-26T06:10:47.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.18330.out 2021-07-26T06:10:47.198 INFO:tasks.workunit.client.0.smithi094.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.18330.out': No such file or directory 2021-07-26T06:10:47.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2021-07-26T06:10:47.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T06:10:47.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T06:10:47.697 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T06:10:47.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2021-07-26T06:10:47.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:10:47.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:10:47.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:10:47.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:10:47.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:10:48.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:10:48.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-07-26T06:10:48.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:10:48.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:10:48.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:10:48.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:10:48.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:10:48.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:10:48.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:10:48.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:10:48.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:10:48.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:10:48.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:10:48.042 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:10:48.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:10:48.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:10:48.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:10:48.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:10:48.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:10:48.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:10:48.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:10:48.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:10:48.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:10:48.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:10:48.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:10:48.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:48.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:48.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:10:48.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:10:48.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:10:48.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-07-26T06:10:48.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-07-26T06:10:48.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-07-26T06:10:48.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-07-26T06:10:48.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:10:48.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:10:48.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:10:48.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:10:48.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:10:48.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:10:48.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:10:48.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:10:48.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:10:48.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:10:48.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:10:48.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:10:48.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:10:48.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:10:48.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:10:48.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:10:48.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:10:48.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:10:48.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:10:48.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:10:48.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:10:48.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:10:48.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:10:48.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:48.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:48.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:10:48.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:10:48.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:10:48.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-07-26T06:10:48.090 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:10:48.090 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:48.090 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:48.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.18330 2021-07-26T06:10:48.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:10:48.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:10:48.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:10:48.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-07-26T06:10:48.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2021-07-26T06:10:48.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2021-07-26T06:10:48.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2021-07-26T06:10:48.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2021-07-26T06:10:48.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2021-07-26T06:10:48.096 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2021-07-26T06:10:48.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2021-07-26T06:10:48.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2021-07-26T06:10:48.098 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:10:48.098 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:10:48.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2021-07-26T06:10:48.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-07-26T06:10:48.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:10:48.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:10:48.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:10:48.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-07-26T06:10:48.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-07-26T06:10:48.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:10:48.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:10:48.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:10:48.445 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:10:48.446 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:48.446 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:48.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:10:48.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:10:48.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:10:48.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:10:48.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:10:48.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:10:48.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:10:48.717 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:10:48.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:10:48.717 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:10:48.717 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:10:48.717 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:10:48.718 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:48.718 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:48.719 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:10:48.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:10:48.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get fsid 2021-07-26T06:10:48.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:10:48.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:10:48.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:10:48.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:10:48.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:10:48.815 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:10:48.815 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:10:48.815 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:10:48.816 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:10:48.816 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:48.816 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:48.816 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:10:48.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:10:48.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get mon_host 2021-07-26T06:10:48.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2021-07-26T06:10:48.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-07-26T06:10:48.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:10:48.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:10:48.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:10:48.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-07-26T06:10:48.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:10:49.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:10:49.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:10:49.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:10:49.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:10:49.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:49.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:49.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:10:49.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:10:49.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:10:49.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-07-26T06:10:49.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2021-07-26T06:10:49.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:10:49.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:10:49.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:10:49.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:10:49.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:10:49.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:10:49.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:10:49.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:10:49.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:10:49.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:10:49.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:10:49.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:10:49.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:10:49.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:10:49.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:10:49.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:10:49.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:10:49.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:10:49.826 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:10:49.826 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:10:49.827 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:10:49.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:10:49.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:10:49.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:10:49.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:10:49.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:10:49.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:10:49.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:10:49.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:10:49.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:10:49.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:10:49.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:10:49.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:10:49.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:10:49.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:10:49.830 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 b2ac31cc-94bd-49d4-89c7-dd5653384a9c 2021-07-26T06:10:49.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2ac31cc-94bd-49d4-89c7-dd5653384a9c 2021-07-26T06:10:49.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b2ac31cc-94bd-49d4-89c7-dd5653384a9c' 2021-07-26T06:10:49.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:10:49.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpUf5gWLu0MhAAbMnVdy/zaHQIk85UDcKeUg== 2021-07-26T06:10:49.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpUf5gWLu0MhAAbMnVdy/zaHQIk85UDcKeUg=="}' 2021-07-26T06:10:49.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2ac31cc-94bd-49d4-89c7-dd5653384a9c -i td/divergent-priors/0/new.json 2021-07-26T06:10:50.248 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:10:50.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2021-07-26T06:10:50.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpUf5gWLu0MhAAbMnVdy/zaHQIk85UDcKeUg== --osd-uuid b2ac31cc-94bd-49d4-89c7-dd5653384a9c 2021-07-26T06:10:50.301 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:50.296+0000 7f3fca531f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:50.303 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:50.296+0000 7f3fca531f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:50.304 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:50.300+0000 7f3fca531f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:50.305 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:50.300+0000 7f3fca531f40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:10:50.305 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:50.300+0000 7f3fca531f40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:10:50.305 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:50.300+0000 7f3fca531f40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:10:50.430 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:50.424+0000 7f3fca531f40 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-07-26T06:10:55.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-07-26T06:10:55.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:10:55.979 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:10:55.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:10:55.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:10:56.499 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:10:56.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:10:56.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:10:56.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:10:56.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:10:56.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:10:56.536 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:56.532+0000 7fdda0fccf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:56.551 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:56.544+0000 7fdda0fccf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:56.552 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:56.548+0000 7fdda0fccf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:10:56.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:10:56.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:10:56.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:10:56.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:10:56.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:10:56.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:56.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:10:56.885 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:10:56.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:56.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:10:57.122 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:57.116+0000 7fdda0fccf40 -1 Falling back to public interface 2021-07-26T06:10:57.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:10:58.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:10:58.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:58.248 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:10:58.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:10:58.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:58.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:10:58.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:10:59.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:10:59.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:10:59.628 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:10:59.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:10:59.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:10:59.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:10:59.669 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:10:59.664+0000 7fdda0fccf40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:11:00.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:01.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:01.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:01.071 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:11:01.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:11:01.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:01.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:11:01.443 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/44927,v1:127.0.0.1:6803/44927] [v2:127.0.0.1:6804/44927,v1:127.0.0.1:6805/44927] exists,up b2ac31cc-94bd-49d4-89c7-dd5653384a9c 2021-07-26T06:11:01.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:11:01.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:11:01.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:11:01.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-07-26T06:11:01.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2021-07-26T06:11:01.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:11:01.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:11:01.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:11:01.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:11:01.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:11:01.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:11:01.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:11:01.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:11:01.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:11:01.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:11:01.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:11:01.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:11:01.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:11:01.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:11:01.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:11:01.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:11:01.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:11:01.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:11:01.453 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:11:01.454 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:01.454 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:11:01.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:11:01.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:11:01.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:11:01.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:11:01.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:11:01.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:11:01.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:11:01.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:11:01.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:11:01.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:11:01.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:11:01.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:11:01.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:11:01.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:11:01.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=64b2a2e5-8ab4-4197-9d59-4b39f363ac4c 2021-07-26T06:11:01.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c' 2021-07-26T06:11:01.458 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c 2021-07-26T06:11:01.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:11:01.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1Uf5gmjppHBAA7HsYoCCW0zmBBolHN6fEHQ== 2021-07-26T06:11:01.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1Uf5gmjppHBAA7HsYoCCW0zmBBolHN6fEHQ=="}' 2021-07-26T06:11:01.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c -i td/divergent-priors/1/new.json 2021-07-26T06:11:02.014 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:11:02.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2021-07-26T06:11:02.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1Uf5gmjppHBAA7HsYoCCW0zmBBolHN6fEHQ== --osd-uuid 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c 2021-07-26T06:11:02.067 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:02.060+0000 7ff2c2384f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:02.077 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:02.072+0000 7ff2c2384f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:02.079 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:02.072+0000 7ff2c2384f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:02.080 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:02.076+0000 7ff2c2384f40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:11:02.080 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:02.076+0000 7ff2c2384f40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:11:02.080 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:02.076+0000 7ff2c2384f40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:11:02.123 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:02.116+0000 7ff2c2384f40 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-07-26T06:11:07.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-07-26T06:11:07.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:11:07.969 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:11:07.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:11:07.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:11:08.450 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:11:08.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:11:08.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:11:08.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:11:08.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:11:08.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:11:08.486 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:08.480+0000 7f800d322f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:08.488 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:08.484+0000 7f800d322f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:08.490 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:08.484+0000 7f800d322f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:08.782 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:08.776+0000 7f800d322f40 -1 Falling back to public interface 2021-07-26T06:11:08.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:11:08.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:11:08.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:11:08.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:11:08.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:11:08.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:08.861 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:11:08.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:11:08.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:08.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:11:09.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:10.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:10.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:10.326 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:11:10.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:11:10.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:10.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:11:10.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:11.423 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:11.420+0000 7f800d322f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:11:11.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:11.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:11.733 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:11:11.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:11:11.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:11.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:11:12.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:13.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:13.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:13.226 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:11:13.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:11:13.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:13.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:11:13.610 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/46333,v1:127.0.0.1:6811/46333] [v2:127.0.0.1:6812/46333,v1:127.0.0.1:6813/46333] exists,up 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c 2021-07-26T06:11:13.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:11:13.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:11:13.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:11:13.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-07-26T06:11:13.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2021-07-26T06:11:13.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:11:13.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:11:13.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:11:13.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:11:13.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:11:13.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:11:13.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:11:13.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:11:13.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:11:13.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:11:13.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:11:13.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:11:13.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:11:13.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:11:13.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:11:13.618 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:11:13.618 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:11:13.618 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:11:13.618 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:11:13.619 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:11:13.619 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:11:13.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:11:13.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:11:13.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:11:13.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:11:13.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:11:13.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:11:13.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:11:13.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:11:13.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:11:13.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:11:13.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:11:13.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:11:13.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:11:13.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:11:13.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8f418200-be0a-4ee5-8381-ae936884bd8d 2021-07-26T06:11:13.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8f418200-be0a-4ee5-8381-ae936884bd8d' 2021-07-26T06:11:13.624 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 8f418200-be0a-4ee5-8381-ae936884bd8d 2021-07-26T06:11:13.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:11:13.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABUv5gggpmJhAA89zWt6hwGrlglz/6EyozMQ== 2021-07-26T06:11:13.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABUv5gggpmJhAA89zWt6hwGrlglz/6EyozMQ=="}' 2021-07-26T06:11:13.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8f418200-be0a-4ee5-8381-ae936884bd8d -i td/divergent-priors/2/new.json 2021-07-26T06:11:14.179 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:11:14.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2021-07-26T06:11:14.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQABUv5gggpmJhAA89zWt6hwGrlglz/6EyozMQ== --osd-uuid 8f418200-be0a-4ee5-8381-ae936884bd8d 2021-07-26T06:11:14.235 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:14.231+0000 7f3274d76f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:14.259 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:14.255+0000 7f3274d76f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:14.262 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:14.259+0000 7f3274d76f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:14.263 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:14.259+0000 7f3274d76f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:11:14.263 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:14.259+0000 7f3274d76f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:11:14.263 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:14.259+0000 7f3274d76f40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:11:14.343 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:14.339+0000 7f3274d76f40 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-07-26T06:11:20.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-07-26T06:11:20.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:11:20.603 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:11:20.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:11:20.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:11:21.091 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:11:21.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:11:21.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:11:21.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:11:21.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:11:21.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:11:21.128 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:21.123+0000 7f2c9fc82f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:21.130 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:21.127+0000 7f2c9fc82f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:21.131 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:21.127+0000 7f2c9fc82f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:11:21.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:11:21.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:11:21.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:11:21.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:11:21.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:11:21.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:21.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:11:21.469 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:11:21.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:21.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:11:21.686 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:21.683+0000 7f2c9fc82f40 -1 Falling back to public interface 2021-07-26T06:11:21.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:22.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:22.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:22.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:11:22.861 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:11:22.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:22.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:11:23.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:24.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:24.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:24.226 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:11:24.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:11:24.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:24.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:11:24.493 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:11:24.491+0000 7f2c9fc82f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:11:24.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:11:25.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:11:25.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:11:25.586 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:11:25.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:11:25.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:11:25.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:11:26.002 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/47739,v1:127.0.0.1:6819/47739] [v2:127.0.0.1:6820/47739,v1:127.0.0.1:6821/47739] exists,up 8f418200-be0a-4ee5-8381-ae936884bd8d 2021-07-26T06:11:26.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:11:26.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:11:26.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:11:26.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2021-07-26T06:11:26.757 INFO:tasks.workunit.client.0.smithi094.stderr:noout is set 2021-07-26T06:11:26.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2021-07-26T06:11:27.551 INFO:tasks.workunit.client.0.smithi094.stderr:noin is set 2021-07-26T06:11:27.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2021-07-26T06:11:28.414 INFO:tasks.workunit.client.0.smithi094.stderr:nodown is set 2021-07-26T06:11:28.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2021-07-26T06:11:28.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T06:11:29.113 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T06:11:29.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:11:30.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2021-07-26T06:11:30.823 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 3 2021-07-26T06:11:30.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2021-07-26T06:11:31.701 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 min_size to 2 2021-07-26T06:11:31.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2021-07-26T06:11:32.627 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 pg_autoscale_mode to on 2021-07-26T06:11:32.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2021-07-26T06:11:32.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:11:32.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:11:33.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:11:33.150 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:11:33.151 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:11:33.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:11:33.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:33.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:11:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T06:11:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T06:11:33.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T06:11:33.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:33.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:11:33.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T06:11:33.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T06:11:33.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2021-07-26T06:11:33.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:33.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:11:33.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509444 2021-07-26T06:11:33.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509444 2021-07-26T06:11:33.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509444' 2021-07-26T06:11:33.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:11:33.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T06:11:33.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:11:33.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:11:33.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T06:11:33.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:11:33.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T06:11:33.630 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836489 2021-07-26T06:11:33.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T06:11:33.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:11:34.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T06:11:34.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:11:34.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T06:11:34.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:11:34.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:11:34.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T06:11:34.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:11:34.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T06:11:34.088 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672968 2021-07-26T06:11:34.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T06:11:34.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:11:34.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672968 2021-07-26T06:11:34.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:11:34.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509444 2021-07-26T06:11:34.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:11:34.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:11:34.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509444 2021-07-26T06:11:34.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:11:34.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509444 2021-07-26T06:11:34.473 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509444 2021-07-26T06:11:34.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-07-26T06:11:34.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:11:34.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509444 -lt 64424509444 2021-07-26T06:11:34.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2021-07-26T06:11:34.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:11:34.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:11:34.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:11:34.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:11:34.920 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:11:34.921 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:11:34.921 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:11:34.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:11:34.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:11:34.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:11:35.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:11:35.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:11:35.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:11:35.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:11:35.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:11:35.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:11:35.492 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:11:35.493 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:11:35.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:11:35.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:35.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:11:35.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836491 2021-07-26T06:11:35.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836491 2021-07-26T06:11:35.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491' 2021-07-26T06:11:35.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:35.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:11:35.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672970 2021-07-26T06:11:35.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672970 2021-07-26T06:11:35.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2021-07-26T06:11:35.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:35.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:11:36.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509447 2021-07-26T06:11:36.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509447 2021-07-26T06:11:36.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509447' 2021-07-26T06:11:36.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:11:36.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836491 2021-07-26T06:11:36.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:11:36.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:11:36.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836491 2021-07-26T06:11:36.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:11:36.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836491 2021-07-26T06:11:36.072 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836491 2021-07-26T06:11:36.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-07-26T06:11:36.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:11:36.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836491 2021-07-26T06:11:36.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:11:36.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672970 2021-07-26T06:11:36.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:11:36.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:11:36.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:11:36.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672970 2021-07-26T06:11:36.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672970 2021-07-26T06:11:36.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-07-26T06:11:36.447 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672970 2021-07-26T06:11:36.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:11:36.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672968 -lt 42949672970 2021-07-26T06:11:36.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:11:37.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:11:37.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:11:38.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672970 2021-07-26T06:11:38.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:11:38.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509447 2021-07-26T06:11:38.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:11:38.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:11:38.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:11:38.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509447 2021-07-26T06:11:38.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509447 2021-07-26T06:11:38.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-07-26T06:11:38.241 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509447 2021-07-26T06:11:38.242 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:11:38.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509447 -lt 64424509447 2021-07-26T06:11:38.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:11:38.608 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:11:38.608 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:11:39.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:11:39.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:11:39.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:11:39.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:11:39.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:11:39.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:11:39.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:11:39.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:11:39.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:11:39.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:11:39.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:11:39.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:11:39.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:11:39.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:11:39.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:11:39.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2021-07-26T06:11:39.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2021-07-26T06:11:40.271 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:11:40.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2021-07-26T06:11:40.281 INFO:tasks.workunit.client.0.smithi094.stdout:primary and soon to be divergent is 1 2021-07-26T06:11:40.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2021-07-26T06:11:40.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:11:40.621 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:11:40.622 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:11:29.777058+0000 0'0 29:13 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:11:40.622 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:11:40.623 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:11:40.624 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:11:40.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2021-07-26T06:11:40.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-07-26T06:11:40.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2021-07-26T06:11:40.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2021-07-26T06:11:40.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-07-26T06:11:40.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2021-07-26T06:11:40.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2021-07-26T06:11:40.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-07-26T06:11:40.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2021-07-26T06:11:40.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2021-07-26T06:11:40.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2021-07-26T06:11:40.636 INFO:tasks.workunit.client.0.smithi094.stdout:writing initial objects 2021-07-26T06:11:40.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2021-07-26T06:11:40.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:40.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2021-07-26T06:11:40.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:40.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2021-07-26T06:11:40.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:40.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2021-07-26T06:11:41.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:41.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2021-07-26T06:11:41.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:41.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2021-07-26T06:11:41.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:41.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2021-07-26T06:11:41.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:41.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2021-07-26T06:11:41.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:41.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2021-07-26T06:11:41.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:41.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2021-07-26T06:11:42.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:42.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2021-07-26T06:11:42.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:42.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2021-07-26T06:11:42.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:42.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2021-07-26T06:11:42.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:42.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2021-07-26T06:11:42.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:42.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2021-07-26T06:11:42.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:42.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2021-07-26T06:11:43.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:43.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2021-07-26T06:11:43.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:43.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2021-07-26T06:11:43.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:43.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2021-07-26T06:11:43.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:43.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2021-07-26T06:11:43.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:43.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2021-07-26T06:11:43.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:43.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2021-07-26T06:11:43.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:43.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2021-07-26T06:11:44.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:44.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2021-07-26T06:11:44.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:44.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2021-07-26T06:11:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2021-07-26T06:11:44.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:44.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2021-07-26T06:11:44.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:44.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2021-07-26T06:11:45.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:45.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2021-07-26T06:11:45.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:45.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2021-07-26T06:11:45.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:45.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2021-07-26T06:11:45.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:45.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2021-07-26T06:11:45.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:45.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2021-07-26T06:11:45.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:45.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2021-07-26T06:11:45.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:45.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2021-07-26T06:11:46.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:46.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2021-07-26T06:11:46.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:46.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2021-07-26T06:11:46.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:46.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2021-07-26T06:11:46.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:46.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2021-07-26T06:11:46.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:46.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2021-07-26T06:11:46.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:46.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2021-07-26T06:11:46.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:46.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2021-07-26T06:11:47.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:47.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2021-07-26T06:11:47.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:47.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2021-07-26T06:11:47.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:47.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2021-07-26T06:11:47.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:47.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2021-07-26T06:11:47.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:47.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2021-07-26T06:11:47.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:47.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2021-07-26T06:11:48.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:48.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2021-07-26T06:11:48.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:48.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2021-07-26T06:11:48.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:48.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2021-07-26T06:11:48.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:48.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2021-07-26T06:11:48.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:48.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2021-07-26T06:11:48.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:48.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2021-07-26T06:11:49.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:49.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2021-07-26T06:11:49.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:49.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2021-07-26T06:11:49.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:49.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2021-07-26T06:11:49.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:49.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2021-07-26T06:11:49.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:49.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2021-07-26T06:11:49.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:49.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2021-07-26T06:11:49.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:49.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2021-07-26T06:11:50.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:50.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2021-07-26T06:11:50.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:50.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2021-07-26T06:11:50.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:50.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2021-07-26T06:11:50.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:50.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2021-07-26T06:11:50.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:50.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2021-07-26T06:11:51.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:51.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2021-07-26T06:11:51.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:51.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2021-07-26T06:11:51.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:51.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2021-07-26T06:11:51.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:51.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2021-07-26T06:11:51.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:51.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2021-07-26T06:11:51.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:51.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2021-07-26T06:11:51.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:51.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2021-07-26T06:11:52.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:52.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2021-07-26T06:11:52.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:52.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2021-07-26T06:11:52.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:52.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2021-07-26T06:11:52.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:52.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2021-07-26T06:11:52.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:52.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2021-07-26T06:11:52.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:52.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2021-07-26T06:11:53.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:53.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2021-07-26T06:11:53.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:53.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2021-07-26T06:11:53.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:53.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2021-07-26T06:11:53.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:53.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2021-07-26T06:11:53.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:53.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2021-07-26T06:11:53.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:53.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2021-07-26T06:11:53.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:53.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2021-07-26T06:11:54.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:54.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2021-07-26T06:11:54.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:54.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2021-07-26T06:11:54.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:54.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2021-07-26T06:11:54.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:54.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2021-07-26T06:11:55.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:55.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2021-07-26T06:11:55.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:55.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2021-07-26T06:11:55.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:55.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2021-07-26T06:11:55.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:55.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2021-07-26T06:11:55.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:55.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2021-07-26T06:11:55.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:55.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2021-07-26T06:11:56.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:56.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2021-07-26T06:11:58.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:58.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2021-07-26T06:11:58.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:58.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2021-07-26T06:11:58.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:58.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2021-07-26T06:11:58.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-07-26T06:11:58.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2021-07-26T06:11:58.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:11:58.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2021-07-26T06:11:58.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:11:58.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:11:58.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:11:58.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:11:58.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:11:58.718 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:11:58.718 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:11:58.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:11:58.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:11:58.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:11:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:11:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:11:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:11:58.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:11:58.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:11:59.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:11:59.237 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:11:59.238 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:11:59.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:11:59.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:59.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:11:59.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836498 2021-07-26T06:11:59.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836498 2021-07-26T06:11:59.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498' 2021-07-26T06:11:59.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:59.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:11:59.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672977 2021-07-26T06:11:59.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672977 2021-07-26T06:11:59.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672977' 2021-07-26T06:11:59.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:11:59.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:11:59.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509453 2021-07-26T06:11:59.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509453 2021-07-26T06:11:59.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509453' 2021-07-26T06:11:59.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:11:59.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836498 2021-07-26T06:11:59.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:11:59.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:11:59.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836498 2021-07-26T06:11:59.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:11:59.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836498 2021-07-26T06:11:59.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2021-07-26T06:11:59.718 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836498 2021-07-26T06:11:59.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:12:00.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836498 -lt 21474836498 2021-07-26T06:12:00.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:12:00.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672977 2021-07-26T06:12:00.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:12:00.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:12:00.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672977 2021-07-26T06:12:00.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:12:00.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672977 2021-07-26T06:12:00.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2021-07-26T06:12:00.226 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672977 2021-07-26T06:12:00.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:12:00.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672977 -lt 42949672977 2021-07-26T06:12:00.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:12:00.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:12:00.892 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509453 2021-07-26T06:12:00.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:12:00.893 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509453 2021-07-26T06:12:00.894 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:12:00.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509453 2021-07-26T06:12:00.897 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509453 2021-07-26T06:12:00.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-07-26T06:12:00.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:12:01.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509453 2021-07-26T06:12:01.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:12:01.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:01.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:01.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 128 == 0 2021-07-26T06:12:01.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:01.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:01.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:01.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:01.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:01.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:01.983 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:02.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=63 2021-07-26T06:12:02.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:02.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:02.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:02.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 63 = 128 2021-07-26T06:12:02.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 63 '!=' -1 2021-07-26T06:12:02.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:12:02.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=63 2021-07-26T06:12:02.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:02.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:12:03.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:03.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:03.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:03.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:03.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:03.004 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:03.004 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:03.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:03.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=63 2021-07-26T06:12:03.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:03.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:03.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:03.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 63 = 128 2021-07-26T06:12:03.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 63 '!=' 63 2021-07-26T06:12:03.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:03.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:03.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:03.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:03.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:03.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:03.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:04.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:04.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:04.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:12:04.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:04.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:12:04.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:04.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:04.591 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:04.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:04.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:04.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:04.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:04.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:04.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=63 2021-07-26T06:12:04.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:04.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:04.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:05.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 63 = 128 2021-07-26T06:12:05.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 63 '!=' 63 2021-07-26T06:12:05.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:05.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:05.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:05.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:05.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:05.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:05.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:06.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:06.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:06.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:12:06.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:06.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:12:06.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:06.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:06.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:06.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:06.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:06.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:06.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:06.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:06.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=63 2021-07-26T06:12:06.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:06.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:06.829 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:07.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 63 = 128 2021-07-26T06:12:07.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 63 '!=' 63 2021-07-26T06:12:07.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:07.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:07.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:07.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:07.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:07.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:07.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:07.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:07.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:07.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:12:07.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:07.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:12:08.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:08.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:08.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:08.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:08.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:08.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:08.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:08.639 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:09.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=128 2021-07-26T06:12:09.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:09.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:09.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:09.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 128 = 128 2021-07-26T06:12:09.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:12:09.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:12:09.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2021-07-26T06:12:09.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T06:12:09.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=existing_1 2021-07-26T06:12:09.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test existing_1 2021-07-26T06:12:09.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:12:09.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.1b 2021-07-26T06:12:09.949 INFO:tasks.workunit.client.0.smithi094.stdout:blackholing osds 0 2 2021-07-26T06:12:09.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2021-07-26T06:12:09.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:12:10.301 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:12:10.301 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.652996+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.301 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.646141+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.302 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 0 0 0 1234 0 0 18 18 active+clean 2021-07-26T06:12:03.500675+0000 34'96 37:129 [1,0,2] 1 [1,0,2] 1 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.302 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.770219+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.302 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:12:02.953712+0000 32'95 37:125 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.302 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.646618+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.303 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 0 0 0 617 0 0 18 18 active+clean 2021-07-26T06:12:05.981531+0000 34'96 37:137 [2,0,1] 2 [2,0,1] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.303 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.018400+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.303 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.646720+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.303 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.656472+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.303 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645034+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.304 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649576+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.304 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.641621+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.304 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:00.897544+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.304 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026059+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.305 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.501018+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.305 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.016900+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.305 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023370+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.305 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.647547+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.306 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645627+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.306 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649729+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.306 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027260+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.306 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659390+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.307 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.648033+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.307 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026282+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.307 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.501578+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.307 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023409+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.307 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659608+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.308 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.771777+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.308 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.505333+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.308 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.620374+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.308 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651056+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.308 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.652533+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.309 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.650625+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.309 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645553+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.309 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.622530+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.309 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.025645+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.310 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.621886+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.310 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:12:01.651234+0000 34'96 37:133 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.310 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654852+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.310 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.623445+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.310 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.644493+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.311 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023693+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.311 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.023558+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.311 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649491+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.311 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:11:59.499344+0000 32'95 36:129 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.311 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022249+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.312 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.652308+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.312 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026255+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.312 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659820+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.312 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654444+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.313 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.650378+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.313 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.620777+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.313 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022870+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.313 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.621496+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.314 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.621578+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.314 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.650608+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.314 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026505+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.314 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.773774+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.315 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.650362+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.315 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654232+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.315 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:12:01.643010+0000 34'96 37:132 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.315 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.532440+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.315 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645483+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.316 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.642558+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.316 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:12:07.455030+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.316 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:12:06.434623+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.316 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651884+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.316 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.728145+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.316 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.029208+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.317 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651729+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.317 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.633696+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.317 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027211+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.317 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.728364+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.318 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.657240+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.318 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.653155+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.318 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659708+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.318 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027421+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.318 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.624033+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.319 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645769+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.319 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.653351+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.319 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.652760+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.319 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.729044+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.319 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.025134+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.320 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.029458+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.320 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023103+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.320 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659856+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.320 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.729365+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.320 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023881+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.321 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.657442+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.321 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651527+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.321 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659989+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.321 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651292+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.322 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.728588+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.322 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.028403+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.322 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654002+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.322 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:12:01.656990+0000 34'96 37:132 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.322 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027645+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.323 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.660041+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.323 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.658850+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.323 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022653+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.323 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.635362+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.324 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.658505+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.324 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654385+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.324 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654649+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.324 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.768266+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.324 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027619+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.325 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027987+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.325 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.727884+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.325 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659055+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.325 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022859+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.325 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.028002+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.326 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.025402+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.326 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654579+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.326 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027478+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.326 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023341+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.327 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027341+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.327 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.779163+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.327 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654191+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.327 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.652802+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.327 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649710+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.328 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.028297+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.328 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.650827+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.328 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.729101+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.328 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.651210+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.328 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:11:57.769024+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.329 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.656690+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.329 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.730087+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:10.329 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:12:10.329 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:12:10.331 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:12:10.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:10.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2021-07-26T06:12:10.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:12:10.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:12:10.332 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:12:10.332 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:10.332 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:12:10.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:12:10.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-07-26T06:12:10.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok config set objectstore_blackhole 1 2021-07-26T06:12:10.391 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:12:10.391 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:12:10.391 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:12:10.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:10.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2021-07-26T06:12:10.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:12:10.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:12:10.401 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:12:10.401 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:10.401 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:12:10.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:12:10.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-07-26T06:12:10.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok config set objectstore_blackhole 1 2021-07-26T06:12:10.482 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:12:10.482 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:12:10.483 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:12:10.489 INFO:tasks.workunit.client.0.smithi094.stdout:create new divergent objects 2021-07-26T06:12:10.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2021-07-26T06:12:10.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2021-07-26T06:12:10.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:12:10.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:12:10.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2021-07-26T06:12:10.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2021-07-26T06:12:10.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2021-07-26T06:12:10.494 INFO:tasks.workunit.client.0.smithi094.stdout:writing divergent objects 2021-07-26T06:12:10.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2021-07-26T06:12:10.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-07-26T06:12:10.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-07-26T06:12:10.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2021-07-26T06:12:10.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2021-07-26T06:12:10.496 INFO:tasks.workunit.client.0.smithi094.stdout:remove divergent objects 2021-07-26T06:12:10.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2021-07-26T06:12:10.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2021-07-26T06:12:10.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-07-26T06:12:10.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2021-07-26T06:12:10.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2021-07-26T06:12:10.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2021-07-26T06:12:10.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-07-26T06:12:10.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2021-07-26T06:12:10.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2021-07-26T06:12:10.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2021-07-26T06:12:10.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2021-07-26T06:12:20.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2021-07-26T06:12:20.511 INFO:tasks.workunit.client.0.smithi094.stderr:rados: no process found 2021-07-26T06:12:20.511 INFO:tasks.workunit.client.0.smithi094.stdout:killing all the osds 2021-07-26T06:12:20.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2021-07-26T06:12:20.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:12:20.875 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:12:20.875 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.652996+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.876 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.646141+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.876 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 0 0 0 1234 0 0 18 18 active+clean 2021-07-26T06:12:03.500675+0000 34'96 37:129 [1,0,2] 1 [1,0,2] 1 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.876 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:12:09.475313+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.876 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:12:02.953712+0000 32'95 37:125 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.876 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.646618+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.877 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 0 0 0 617 0 0 18 18 active+clean 2021-07-26T06:12:05.981531+0000 34'96 37:137 [2,0,1] 2 [2,0,1] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.877 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 0 0 0 1851 0 0 17 17 active+clean 2021-07-26T06:12:07.948142+0000 32'95 37:125 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.877 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.646720+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.877 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.656472+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.877 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645034+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.878 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649576+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.878 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 2 0 0 0 0 458 0 0 18 18 active+clean 2021-07-26T06:12:01.641621+0000 37'96 37:133 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.878 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:00.897544+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.878 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026059+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.879 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.501018+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.879 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:12:06.933955+0000 32'95 37:125 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:12:20.879 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023370+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.879 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.647547+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.879 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645627+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.880 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649729+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.880 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027260+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.880 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659390+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.880 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.648033+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.880 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026282+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.881 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.501578+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.881 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023409+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.881 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659608+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.881 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.771777+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.881 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.505333+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.882 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.620374+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.882 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651056+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.882 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.652533+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.882 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.650625+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.882 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645553+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.883 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.622530+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.883 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.025645+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.883 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.621886+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.883 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:12:01.651234+0000 34'96 37:133 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.883 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654852+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.884 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.623445+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.884 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.644493+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.884 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023693+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.884 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.023558+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.884 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649491+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.885 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:11:59.499344+0000 32'95 36:129 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.885 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022249+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.885 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 0 0 0 0 0 0 0 0 18 18 active+clean 2021-07-26T06:12:01.652308+0000 37'96 37:133 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.885 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026255+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.886 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:12:12.806592+0000 32'95 37:125 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:12.806402+0000 32'95 2021-07-26T06:12:12.806402+0000 0 2021-07-26T06:12:20.886 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 0 0 0 0 0 0 0 0 18 18 active+clean 2021-07-26T06:12:01.654444+0000 37'96 37:121 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.886 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.650378+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.886 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.620777+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.886 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022870+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.887 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.621496+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.887 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:12:14.634180+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:14.633991+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.887 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:12:07.815282+0000 32'95 37:125 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.887 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.026505+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.887 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.773774+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.888 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.650362+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.888 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 0 0 0 601 0 0 18 18 active+clean 2021-07-26T06:12:01.654232+0000 37'96 37:121 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.888 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:12:01.643010+0000 34'96 37:132 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.888 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.532440+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.888 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645483+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.889 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.642558+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.889 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:12:07.455030+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.889 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:12:06.434623+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.889 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651884+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.890 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.728145+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.890 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.029208+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.890 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651729+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.890 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.633696+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.890 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027211+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.891 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.728364+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.891 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.657240+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.891 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.653155+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.891 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659708+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.892 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027421+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.892 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.624033+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.892 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.645769+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.892 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.653351+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.892 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.652760+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.893 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.729044+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.893 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.025134+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.893 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.029458+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.893 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023103+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.893 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659856+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.894 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.729365+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.894 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023881+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.894 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.657442+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.894 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651527+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.894 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659989+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.895 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.651292+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.895 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 2 0 0 0 0 458 0 0 18 18 active+clean 2021-07-26T06:12:01.728588+0000 37'96 37:121 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.895 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.028403+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.895 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654002+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.895 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:12:01.656990+0000 34'96 37:132 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.896 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027645+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.896 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:12:13.844049+0000 32'95 37:125 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:13.843874+0000 32'95 2021-07-26T06:12:13.843874+0000 0 2021-07-26T06:12:20.896 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.658850+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.896 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022653+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.896 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.635362+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.897 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.658505+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.897 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654385+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.897 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654649+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.897 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.768266+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.897 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027619+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.898 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 601 0 0 18 18 active+clean 2021-07-26T06:12:01.027987+0000 37'96 37:121 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.898 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:12:14.812333+0000 32'95 37:125 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:14.812154+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.898 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.659055+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.898 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.022859+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.899 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.028002+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.899 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.025402+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.899 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654579+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.899 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027478+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.899 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.023341+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.900 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.027341+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.900 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:11:57.779163+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.900 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.654191+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.900 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.652802+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.901 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.649710+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.901 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:12:01.028297+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.901 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.650827+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.901 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.729101+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.901 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.651210+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.902 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:11:57.769024+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.902 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.656690+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.902 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:12:01.730087+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:20.902 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:12:20.903 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:12:20.904 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:12:20.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2021-07-26T06:12:20.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:12:20.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:12:20.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:12:20.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:12:20.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:12:21.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:12:21.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-07-26T06:12:21.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2021-07-26T06:12:21.531 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:12:21.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-07-26T06:12:21.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2021-07-26T06:12:22.111 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T06:12:22.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-07-26T06:12:22.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2021-07-26T06:12:22.724 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already down. 2021-07-26T06:12:22.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:22.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2021-07-26T06:12:23.289 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T06:12:23.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:23.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2021-07-26T06:12:23.847 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already out. 2021-07-26T06:12:23.857 INFO:tasks.workunit.client.0.smithi094.stdout:bringing up non_divergent 0 2 2021-07-26T06:12:23.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2021-07-26T06:12:23.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:12:24.213 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:12:24.213 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.652996+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.213 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.646141+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.213 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 0 0 0 1234 0 0 18 18 stale+active+clean 2021-07-26T06:12:03.500675+0000 34'96 37:129 [1,0,2] 1 [1,0,2] 1 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.214 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 0 0 0 0 0 0 0 0 17 17 stale+active+clean 2021-07-26T06:12:09.475313+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.214 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 0 0 0 617 0 0 17 17 stale+active+clean 2021-07-26T06:12:02.953712+0000 32'95 37:125 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.214 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.646618+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.214 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 0 0 0 617 0 0 18 18 stale+active+clean 2021-07-26T06:12:05.981531+0000 34'96 37:137 [2,0,1] 2 [2,0,1] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 0 0 0 1851 0 0 17 17 stale+active+clean 2021-07-26T06:12:07.948142+0000 32'95 37:125 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.646720+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.656472+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.645034+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.649576+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 2 0 0 0 0 458 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.641621+0000 37'96 37:133 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:00.897544+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.026059+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.501018+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 stale+active+clean 2021-07-26T06:12:06.933955+0000 32'95 37:125 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:12:24.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.023370+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.647547+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.645627+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.649729+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.027260+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.659390+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.648033+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.026282+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.501578+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.023409+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.659608+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.771777+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.505333+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.620374+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.651056+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.652533+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.650625+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.645553+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.622530+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.025645+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.621886+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 0 0 0 1075 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.651234+0000 34'96 37:133 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.654852+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.623445+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.644493+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.023693+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.023558+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.649491+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 17 17 stale+active+clean 2021-07-26T06:11:59.499344+0000 32'95 36:129 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.022249+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 0 0 0 0 0 0 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.652308+0000 37'96 37:133 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.026255+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 0 0 0 617 0 0 17 17 stale+active+clean 2021-07-26T06:12:12.806592+0000 32'95 37:125 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:12.806402+0000 32'95 2021-07-26T06:12:12.806402+0000 0 2021-07-26T06:12:24.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 0 0 0 0 0 0 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.654444+0000 37'96 37:121 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.650378+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.620777+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.022870+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.621496+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 0 0 0 617 0 0 17 17 stale+active+clean 2021-07-26T06:12:14.634180+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:14.633991+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 stale+active+clean 2021-07-26T06:12:07.815282+0000 32'95 37:125 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.026505+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.773774+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.650362+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 0 0 0 601 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.654232+0000 37'96 37:121 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 0 0 0 1075 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.643010+0000 34'96 37:132 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.532440+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.645483+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.642558+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 0 0 0 0 0 0 0 0 17 17 stale+active+clean 2021-07-26T06:12:07.455030+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 0 0 0 617 0 0 17 17 stale+active+clean 2021-07-26T06:12:06.434623+0000 32'95 37:128 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.651884+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.728145+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.029208+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.651729+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.633696+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.027211+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.728364+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.657240+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.653155+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.659708+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.027421+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.624033+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.645769+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.231 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.653351+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.231 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.652760+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.231 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.729044+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.231 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.025134+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.029458+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.023103+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.659856+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.729365+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.023881+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.657442+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.651527+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.659989+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.651292+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 2 0 0 0 0 458 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.728588+0000 37'96 37:121 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.028403+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.654002+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 0 0 0 1075 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.656990+0000 34'96 37:132 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.027645+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 1 0 0 0 0 617 0 0 17 17 stale+active+clean 2021-07-26T06:12:13.844049+0000 32'95 37:125 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:13.843874+0000 32'95 2021-07-26T06:12:13.843874+0000 0 2021-07-26T06:12:24.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.658850+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.022653+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.635362+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.658505+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.654385+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.654649+0000 32'95 37:120 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.768266+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.027619+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 601 0 0 18 18 stale+active+clean 2021-07-26T06:12:01.027987+0000 37'96 37:121 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 0 0 0 17 17 stale+active+clean 2021-07-26T06:12:14.812333+0000 32'95 37:125 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:14.812154+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.659055+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.022859+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.238 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.028002+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.238 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.025402+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.238 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.654579+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.238 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.027478+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.238 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.023341+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.239 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.027341+0000 32'95 36:120 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.239 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.779163+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.239 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.654191+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.239 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.652802+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.240 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.649710+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.240 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.028297+0000 32'95 36:120 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.240 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.650827+0000 32'95 37:132 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.240 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.729101+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.241 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.651210+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.241 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:11:57.769024+0000 32'95 36:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.241 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.656690+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.241 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 stale+active+clean 2021-07-26T06:12:01.730087+0000 32'95 37:120 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:12:24.241 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:12:24.242 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:12:24.246 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:12:24.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:24.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2021-07-26T06:12:24.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:12:24.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:12:24.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:12:24.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:12:24.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:12:24.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:12:24.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:24.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:24.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:24.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:12:24.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:12:24.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:24.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:24.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:12:24.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:12:24.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:12:24.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:24.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:24.251 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:24.251 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:24.251 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:12:24.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:12:24.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:12:24.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:24.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:12:24.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:12:24.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:24.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:24.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:24.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:12:24.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:12:24.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:12:24.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:12:24.254 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:12:24.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:24.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/0/whoami 2021-07-26T06:12:24.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:12:24.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:12:24.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:12:24.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:12:24.268 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:24.262+0000 7f2eb2a16f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:24.270 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:24.266+0000 7f2eb2a16f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:24.272 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:24.270+0000 7f2eb2a16f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:24.571 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:24.566+0000 7f2eb2a16f40 -1 Falling back to public interface 2021-07-26T06:12:24.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:12:24.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:12:24.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:12:24.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:12:24.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:12:24.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:24.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:12:24.602 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:12:24.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:24.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:12:24.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:25.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:25.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:25.992 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:12:25.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:12:25.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:25.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:12:26.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:27.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:27.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:27.366 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:12:27.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:27.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:27.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:12:27.755 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:27.750+0000 7f2eb2a16f40 -1 osd.0 37 log_to_monitors {default=true} 2021-07-26T06:12:27.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:28.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:28.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:28.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:28.782 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:12:28.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:28.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:12:29.177 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up out weight 0 up_from 44 up_thru 35 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/52124,v1:127.0.0.1:6803/52124] [v2:127.0.0.1:6804/52124,v1:127.0.0.1:6805/52124] exists,up b2ac31cc-94bd-49d4-89c7-dd5653384a9c 2021-07-26T06:12:29.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:29.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:29.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:29.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:29.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2021-07-26T06:12:29.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:12:29.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:12:29.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:12:29.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:12:29.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:12:29.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:12:29.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:12:29.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:12:29.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:12:29.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:12:29.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:12:29.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:12:29.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:12:29.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:12:29.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:12:29.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:12:29.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:12:29.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:12:29.185 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:12:29.185 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:12:29.185 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:12:29.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:12:29.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:12:29.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:12:29.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:12:29.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:12:29.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:12:29.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:12:29.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:29.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:12:29.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:12:29.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:12:29.189 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:12:29.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:12:29.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:12:29.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/2/whoami 2021-07-26T06:12:29.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:12:29.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:12:29.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:12:29.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:12:29.218 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:29.214+0000 7f4b1a754f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:29.260 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:29.258+0000 7f4b1a754f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:29.262 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:29.258+0000 7f4b1a754f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:12:29.547 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:29.542+0000 7f4b1a754f40 -1 Falling back to public interface 2021-07-26T06:12:29.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:12:29.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:12:29.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:12:29.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:12:29.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:12:29.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:29.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:12:29.549 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:12:29.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:29.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:29.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:30.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:30.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:30.924 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:12:30.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:12:30.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:30.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:31.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:32.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:32.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:32.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:12:32.304 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:12:32.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:32.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:32.652 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:12:32.650+0000 7f4b1a754f40 -1 osd.2 37 log_to_monitors {default=true} 2021-07-26T06:12:32.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:12:33.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:12:33.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:12:33.692 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:12:33.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:12:33.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:12:33.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:12:34.094 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up out weight 0 up_from 46 up_thru 35 down_at 40 last_clean_interval [15,37) [v2:127.0.0.1:6810/52704,v1:127.0.0.1:6811/52704] [v2:127.0.0.1:6812/52704,v1:127.0.0.1:6813/52704] exists,up 8f418200-be0a-4ee5-8381-ae936884bd8d 2021-07-26T06:12:34.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:12:34.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:12:34.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:12:34.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:34.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2021-07-26T06:12:34.770 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already in. 2021-07-26T06:12:34.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:12:34.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2021-07-26T06:12:36.320 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already in. 2021-07-26T06:12:36.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:12:36.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2021-07-26T06:12:36.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:12:36.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:12:36.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:12:36.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:12:36.335 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:12:36.336 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:12:36.336 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:12:36.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:12:36.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:12:36.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:12:36.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:12:36.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:12:36.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:12:36.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:12:36.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:12:36.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:12:36.935 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:12:36.935 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:12:36.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:12:36.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:12:36.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:12:37.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=188978561028 2021-07-26T06:12:37.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 188978561028 2021-07-26T06:12:37.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-188978561028' 2021-07-26T06:12:37.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:12:37.190 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:12:37.302 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-07-26T06:12:37.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T06:12:37.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T06:12:37.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T06:12:37.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:12:37.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:12:37.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495619 2021-07-26T06:12:37.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495619 2021-07-26T06:12:37.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-188978561028 2-197568495619' 2021-07-26T06:12:37.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:12:37.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-188978561028 2021-07-26T06:12:37.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:12:37.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:12:37.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-188978561028 2021-07-26T06:12:37.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:12:37.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=188978561028 2021-07-26T06:12:37.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 188978561028' 2021-07-26T06:12:37.525 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 188978561028 2021-07-26T06:12:37.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:12:37.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 188978561026 -lt 188978561028 2021-07-26T06:12:37.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:12:38.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:12:38.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:12:39.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 188978561028 -lt 188978561028 2021-07-26T06:12:39.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:12:39.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-197568495619 2021-07-26T06:12:39.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:12:39.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:12:39.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-197568495619 2021-07-26T06:12:39.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:12:39.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495619 2021-07-26T06:12:39.345 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 197568495619 2021-07-26T06:12:39.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2021-07-26T06:12:39.346 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:12:39.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495619 -lt 197568495619 2021-07-26T06:12:39.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:12:39.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:39.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:40.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 128 == 0 2021-07-26T06:12:40.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:40.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:40.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:40.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:40.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:40.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:40.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:40.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:12:40.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:40.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:40.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:41.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:12:41.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:12:41.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:12:41.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:12:41.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:41.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:12:41.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:41.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:41.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:41.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:41.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:41.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:41.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:41.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:41.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:12:41.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:41.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:41.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:42.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:12:42.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:12:42.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:42.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:42.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:42.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:42.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:42.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:42.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:42.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:12:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:12:42.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:42.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:42.841 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:42.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:42.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:42.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:42.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:42.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:43.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:12:43.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:43.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:43.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:43.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:12:43.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:12:43.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:43.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:43.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:43.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:43.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:43.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:43.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:44.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:44.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:44.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:12:44.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:44.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:12:44.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:44.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:44.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:44.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:44.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:44.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:44.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:44.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:45.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:12:45.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:45.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:45.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:45.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:12:45.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:12:45.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:45.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:45.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:45.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:45.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:45.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:45.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:45.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:45.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:45.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:12:45.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:45.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:12:46.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:46.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:46.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:46.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:46.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:46.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:46.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:46.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:47.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:12:47.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:47.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:47.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:47.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:12:47.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:12:47.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:47.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:47.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:47.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:47.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:47.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:47.679 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:48.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:48.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:48.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 4 >= 8 )) 2021-07-26T06:12:48.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:48.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 1.6 2021-07-26T06:12:49.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:49.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:49.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:49.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:49.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:49.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:49.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:49.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:50.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:12:50.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:50.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:50.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:50.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:12:50.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:12:50.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:50.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:50.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:50.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:50.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:50.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:50.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:51.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:51.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:51.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 5 >= 8 )) 2021-07-26T06:12:51.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:51.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 3.2 2021-07-26T06:12:54.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:12:54.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:12:54.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:12:54.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:12:54.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:12:54.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:12:54.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:12:54.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:12:54.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:12:54.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:12:54.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:12:54.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:12:55.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:12:55.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:12:55.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:12:55.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:12:55.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:12:55.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:12:55.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:12:55.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:12:55.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:12:55.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:12:55.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:12:55.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 6 >= 8 )) 2021-07-26T06:12:55.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:12:55.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 6.4 2021-07-26T06:13:02.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:02.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:02.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:02.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:02.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:02.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:02.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:02.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:02.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:02.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:02.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:02.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:03.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:03.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:03.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:03.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:03.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:03.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:03.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:03.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:03.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:03.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:03.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:03.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 7 >= 8 )) 2021-07-26T06:13:03.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:03.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 7.3 2021-07-26T06:13:11.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:11.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:11.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:11.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:11.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:11.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:11.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:11.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:11.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:11.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:11.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:11.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:11.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:11.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:11.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:11.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:11.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:11.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:11.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:12.000 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:12.000 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:12.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:12.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:12.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 8 >= 8 )) 2021-07-26T06:13:12.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph report 2021-07-26T06:13:12.968 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:13:12.969 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_fingerprint": "2e823565-04be-471a-8f59-734ee1c21e89", 2021-07-26T06:13:12.969 INFO:tasks.workunit.client.0.smithi094.stdout: "version": "17.0.0-6358-g197e564f", 2021-07-26T06:13:12.969 INFO:tasks.workunit.client.0.smithi094.stdout: "commit": "197e564fdaee12596de1dc8f3253d1704ae66d5d", 2021-07-26T06:13:12.969 INFO:tasks.workunit.client.0.smithi094.stdout: "timestamp": "2021-07-26T06:13:12.965991+0000", 2021-07-26T06:13:12.970 INFO:tasks.workunit.client.0.smithi094.stdout: "tag": "", 2021-07-26T06:13:12.970 INFO:tasks.workunit.client.0.smithi094.stdout: "health": { 2021-07-26T06:13:12.970 INFO:tasks.workunit.client.0.smithi094.stdout: "status": "HEALTH_WARN", 2021-07-26T06:13:12.970 INFO:tasks.workunit.client.0.smithi094.stdout: "checks": { 2021-07-26T06:13:12.970 INFO:tasks.workunit.client.0.smithi094.stdout: "OSDMAP_FLAGS": { 2021-07-26T06:13:12.971 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:12.971 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:12.971 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "nodown,noout,noin flag(s) set", 2021-07-26T06:13:12.971 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 17 2021-07-26T06:13:12.972 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.972 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [], 2021-07-26T06:13:12.972 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:12.972 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.973 INFO:tasks.workunit.client.0.smithi094.stdout: "OSD_DOWN": { 2021-07-26T06:13:12.973 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:12.973 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:12.973 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 osds down", 2021-07-26T06:13:12.973 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:13:12.974 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.974 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:13:12.974 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.974 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "osd.1 (root=default,host=smithi094) is down" 2021-07-26T06:13:12.975 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:12.975 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:12.975 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:12.975 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.975 INFO:tasks.workunit.client.0.smithi094.stdout: "PG_DEGRADED": { 2021-07-26T06:13:12.976 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:12.976 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:12.976 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "Degraded data redundancy: 99/297 objects degraded (33.333%), 93 pgs degraded", 2021-07-26T06:13:12.976 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 93 2021-07-26T06:13:12.977 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.977 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:13:12.977 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.977 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.978 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.978 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.978 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.978 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.979 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.979 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.979 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.979 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.980 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.980 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.980 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.981 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.981 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.982 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.982 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.983 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.983 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.983 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.984 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.984 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.985 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.985 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.985 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.986 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.986 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.987 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.987 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.988 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.988 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.988 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.989 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.989 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.989 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.990 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.990 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.990 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.990 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.991 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.991 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.991 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.991 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.992 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.992 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.992 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.992 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.992 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.993 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.993 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.993 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.993 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.993 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.994 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.994 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.994 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.994 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.995 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.995 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.995 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.995 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.995 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.996 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.996 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.996 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.996 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.997 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.997 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.997 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.997 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.997 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.998 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.998 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.998 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:12.998 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.999 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.999 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.20 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:12.999 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:12.999 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:12.999 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.21 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.000 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.000 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.000 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.22 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.000 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.001 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.001 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.23 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.001 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.001 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.002 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.24 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.002 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.002 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.002 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.25 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.002 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.003 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.003 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.26 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.003 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.003 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.004 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.27 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.004 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.004 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.004 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.28 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.005 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.005 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.005 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.29 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.005 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.006 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.006 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2b is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.006 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.006 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.006 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2c is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.007 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.007 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.007 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2d is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.007 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.008 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.008 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2e is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.008 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.008 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.008 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2f is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.009 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.009 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.009 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.30 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.009 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.010 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.010 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.31 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.010 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.010 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.010 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.32 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.011 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.011 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.011 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.33 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.011 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.011 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.012 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.34 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.012 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.012 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.012 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.35 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.013 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.013 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.013 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.36 is active+undersized+degraded, acting [2,0]" 2021-07-26T06:13:13.013 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.013 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.014 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.37 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.014 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.014 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.014 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.38 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.015 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.015 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.015 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.39 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.015 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.015 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.016 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.3a is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:13.016 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.016 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.016 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:13.016 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.017 INFO:tasks.workunit.client.0.smithi094.stdout: "POOL_APP_NOT_ENABLED": { 2021-07-26T06:13:13.017 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:13.017 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:13.017 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 pool(s) do not have an application enabled", 2021-07-26T06:13:13.018 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:13:13.018 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.018 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:13:13.018 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.019 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "application not enabled on pool 'test'" 2021-07-26T06:13:13.019 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.019 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.019 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-07-26T06:13:13.020 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.020 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.020 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:13.020 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.021 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.021 INFO:tasks.workunit.client.0.smithi094.stdout: "mutes": [] 2021-07-26T06:13:13.021 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.021 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_first_committed": 1, 2021-07-26T06:13:13.021 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_last_committed": 1, 2021-07-26T06:13:13.022 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap": { 2021-07-26T06:13:13.022 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:13:13.022 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:13:13.022 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:10:48.128736Z", 2021-07-26T06:13:13.023 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:10:48.128736Z", 2021-07-26T06:13:13.023 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release": 17, 2021-07-26T06:13:13.023 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release_name": "quincy", 2021-07-26T06:13:13.023 INFO:tasks.workunit.client.0.smithi094.stdout: "election_strategy": 1, 2021-07-26T06:13:13.023 INFO:tasks.workunit.client.0.smithi094.stdout: "disallowed_leaders: ": "", 2021-07-26T06:13:13.024 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": false, 2021-07-26T06:13:13.024 INFO:tasks.workunit.client.0.smithi094.stdout: "features": { 2021-07-26T06:13:13.024 INFO:tasks.workunit.client.0.smithi094.stdout: "persistent": [ 2021-07-26T06:13:13.024 INFO:tasks.workunit.client.0.smithi094.stdout: "kraken", 2021-07-26T06:13:13.025 INFO:tasks.workunit.client.0.smithi094.stdout: "luminous", 2021-07-26T06:13:13.025 INFO:tasks.workunit.client.0.smithi094.stdout: "mimic", 2021-07-26T06:13:13.025 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap-prune", 2021-07-26T06:13:13.025 INFO:tasks.workunit.client.0.smithi094.stdout: "nautilus", 2021-07-26T06:13:13.026 INFO:tasks.workunit.client.0.smithi094.stdout: "octopus", 2021-07-26T06:13:13.026 INFO:tasks.workunit.client.0.smithi094.stdout: "pacific", 2021-07-26T06:13:13.026 INFO:tasks.workunit.client.0.smithi094.stdout: "elector-pinging", 2021-07-26T06:13:13.026 INFO:tasks.workunit.client.0.smithi094.stdout: "quincy" 2021-07-26T06:13:13.026 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.027 INFO:tasks.workunit.client.0.smithi094.stdout: "optional": [] 2021-07-26T06:13:13.027 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.027 INFO:tasks.workunit.client.0.smithi094.stdout: "mons": [ 2021-07-26T06:13:13.027 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.027 INFO:tasks.workunit.client.0.smithi094.stdout: "rank": 0, 2021-07-26T06:13:13.028 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "a", 2021-07-26T06:13:13.028 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:13.028 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.028 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.029 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.029 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115", 2021-07-26T06:13:13.029 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 0 2021-07-26T06:13:13.029 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.029 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.030 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.030 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115/0", 2021-07-26T06:13:13.030 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:7115/0", 2021-07-26T06:13:13.030 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 0, 2021-07-26T06:13:13.031 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 0, 2021-07-26T06:13:13.031 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_location": "{}" 2021-07-26T06:13:13.031 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.031 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.031 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.032 INFO:tasks.workunit.client.0.smithi094.stdout: "quorum": [ 2021-07-26T06:13:13.032 INFO:tasks.workunit.client.0.smithi094.stdout: 0 2021-07-26T06:13:13.032 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.032 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap": { 2021-07-26T06:13:13.032 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 53, 2021-07-26T06:13:13.033 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:13:13.033 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:10:49.009557+0000", 2021-07-26T06:13:13.033 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:12:36.913606+0000", 2021-07-26T06:13:13.033 INFO:tasks.workunit.client.0.smithi094.stdout: "last_up_change": "2021-07-26T06:12:32.988643+0000", 2021-07-26T06:13:13.034 INFO:tasks.workunit.client.0.smithi094.stdout: "last_in_change": "2021-07-26T06:12:35.540674+0000", 2021-07-26T06:13:13.034 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-07-26T06:13:13.034 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_num": 5800384, 2021-07-26T06:13:13.034 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_set": [ 2021-07-26T06:13:13.034 INFO:tasks.workunit.client.0.smithi094.stdout: "nodown", 2021-07-26T06:13:13.035 INFO:tasks.workunit.client.0.smithi094.stdout: "noin", 2021-07-26T06:13:13.035 INFO:tasks.workunit.client.0.smithi094.stdout: "noout", 2021-07-26T06:13:13.035 INFO:tasks.workunit.client.0.smithi094.stdout: "pglog_hardlimit", 2021-07-26T06:13:13.035 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snapdirs", 2021-07-26T06:13:13.036 INFO:tasks.workunit.client.0.smithi094.stdout: "recovery_deletes", 2021-07-26T06:13:13.036 INFO:tasks.workunit.client.0.smithi094.stdout: "sortbitwise" 2021-07-26T06:13:13.036 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.036 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_version": 9, 2021-07-26T06:13:13.036 INFO:tasks.workunit.client.0.smithi094.stdout: "full_ratio": 0.99000000953674316, 2021-07-26T06:13:13.037 INFO:tasks.workunit.client.0.smithi094.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-07-26T06:13:13.037 INFO:tasks.workunit.client.0.smithi094.stdout: "nearfull_ratio": 0.85000002384185791, 2021-07-26T06:13:13.037 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_snapshot": "", 2021-07-26T06:13:13.037 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_max": 1, 2021-07-26T06:13:13.038 INFO:tasks.workunit.client.0.smithi094.stdout: "max_osd": 3, 2021-07-26T06:13:13.038 INFO:tasks.workunit.client.0.smithi094.stdout: "require_min_compat_client": "luminous", 2021-07-26T06:13:13.038 INFO:tasks.workunit.client.0.smithi094.stdout: "min_compat_client": "jewel", 2021-07-26T06:13:13.038 INFO:tasks.workunit.client.0.smithi094.stdout: "require_osd_release": "quincy", 2021-07-26T06:13:13.038 INFO:tasks.workunit.client.0.smithi094.stdout: "pools": [ 2021-07-26T06:13:13.039 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.039 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:13:13.039 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_name": "test", 2021-07-26T06:13:13.039 INFO:tasks.workunit.client.0.smithi094.stdout: "create_time": "2021-07-26T06:11:28.711595+0000", 2021-07-26T06:13:13.040 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": 1, 2021-07-26T06:13:13.040 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_names": "hashpspool", 2021-07-26T06:13:13.040 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:13:13.040 INFO:tasks.workunit.client.0.smithi094.stdout: "size": 3, 2021-07-26T06:13:13.041 INFO:tasks.workunit.client.0.smithi094.stdout: "min_size": 2, 2021-07-26T06:13:13.041 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_rule": 0, 2021-07-26T06:13:13.041 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_count": 0, 2021-07-26T06:13:13.041 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_target": 0, 2021-07-26T06:13:13.042 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_barrier": 0, 2021-07-26T06:13:13.042 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-07-26T06:13:13.042 INFO:tasks.workunit.client.0.smithi094.stdout: "object_hash": 2, 2021-07-26T06:13:13.042 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_autoscale_mode": "on", 2021-07-26T06:13:13.042 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num": 128, 2021-07-26T06:13:13.043 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num": 128, 2021-07-26T06:13:13.043 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num_target": 128, 2021-07-26T06:13:13.043 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_target": 128, 2021-07-26T06:13:13.043 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_pending": 128, 2021-07-26T06:13:13.043 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pg_merge_meta": { 2021-07-26T06:13:13.044 INFO:tasks.workunit.client.0.smithi094.stdout: "source_pgid": "0.0", 2021-07-26T06:13:13.044 INFO:tasks.workunit.client.0.smithi094.stdout: "ready_epoch": 0, 2021-07-26T06:13:13.044 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_started": 0, 2021-07-26T06:13:13.045 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": 0, 2021-07-26T06:13:13.045 INFO:tasks.workunit.client.0.smithi094.stdout: "source_version": "0'0", 2021-07-26T06:13:13.045 INFO:tasks.workunit.client.0.smithi094.stdout: "target_version": "0'0" 2021-07-26T06:13:13.045 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.046 INFO:tasks.workunit.client.0.smithi094.stdout: "last_change": "35", 2021-07-26T06:13:13.046 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend": "0", 2021-07-26T06:13:13.046 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_prenautilus": "0", 2021-07-26T06:13:13.046 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_preluminous": "33", 2021-07-26T06:13:13.047 INFO:tasks.workunit.client.0.smithi094.stdout: "auid": 0, 2021-07-26T06:13:13.047 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_mode": "selfmanaged", 2021-07-26T06:13:13.048 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_seq": 0, 2021-07-26T06:13:13.048 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_epoch": 0, 2021-07-26T06:13:13.049 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_snaps": [], 2021-07-26T06:13:13.049 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps": "[]", 2021-07-26T06:13:13.049 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_bytes": 0, 2021-07-26T06:13:13.050 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_objects": 0, 2021-07-26T06:13:13.050 INFO:tasks.workunit.client.0.smithi094.stdout: "tiers": [], 2021-07-26T06:13:13.051 INFO:tasks.workunit.client.0.smithi094.stdout: "tier_of": -1, 2021-07-26T06:13:13.051 INFO:tasks.workunit.client.0.smithi094.stdout: "read_tier": -1, 2021-07-26T06:13:13.052 INFO:tasks.workunit.client.0.smithi094.stdout: "write_tier": -1, 2021-07-26T06:13:13.052 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_mode": "none", 2021-07-26T06:13:13.053 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_bytes": 0, 2021-07-26T06:13:13.053 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_objects": 0, 2021-07-26T06:13:13.053 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-07-26T06:13:13.054 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-07-26T06:13:13.054 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_full_ratio_micro": 800000, 2021-07-26T06:13:13.055 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_flush_age": 0, 2021-07-26T06:13:13.055 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_evict_age": 0, 2021-07-26T06:13:13.056 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profile": "", 2021-07-26T06:13:13.056 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_params": { 2021-07-26T06:13:13.056 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "none" 2021-07-26T06:13:13.056 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.056 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_period": 0, 2021-07-26T06:13:13.057 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_count": 0, 2021-07-26T06:13:13.057 INFO:tasks.workunit.client.0.smithi094.stdout: "use_gmt_hitset": true, 2021-07-26T06:13:13.057 INFO:tasks.workunit.client.0.smithi094.stdout: "min_read_recency_for_promote": 0, 2021-07-26T06:13:13.057 INFO:tasks.workunit.client.0.smithi094.stdout: "min_write_recency_for_promote": 0, 2021-07-26T06:13:13.057 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_grade_decay_rate": 0, 2021-07-26T06:13:13.058 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_search_last_n": 0, 2021-07-26T06:13:13.058 INFO:tasks.workunit.client.0.smithi094.stdout: "grade_table": [], 2021-07-26T06:13:13.059 INFO:tasks.workunit.client.0.smithi094.stdout: "stripe_width": 0, 2021-07-26T06:13:13.059 INFO:tasks.workunit.client.0.smithi094.stdout: "expected_num_objects": 0, 2021-07-26T06:13:13.059 INFO:tasks.workunit.client.0.smithi094.stdout: "fast_read": false, 2021-07-26T06:13:13.060 INFO:tasks.workunit.client.0.smithi094.stdout: "options": {}, 2021-07-26T06:13:13.060 INFO:tasks.workunit.client.0.smithi094.stdout: "application_metadata": {} 2021-07-26T06:13:13.060 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.060 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.060 INFO:tasks.workunit.client.0.smithi094.stdout: "osds": [ 2021-07-26T06:13:13.061 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.061 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:13.061 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "b2ac31cc-94bd-49d4-89c7-dd5653384a9c", 2021-07-26T06:13:13.061 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:13:13.061 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:13:13.062 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:13:13.062 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:13:13.062 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 5, 2021-07-26T06:13:13.062 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 37, 2021-07-26T06:13:13.063 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 44, 2021-07-26T06:13:13.063 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 50, 2021-07-26T06:13:13.063 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 38, 2021-07-26T06:13:13.063 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:13:13.064 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:13.064 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.064 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.064 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.064 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6802", 2021-07-26T06:13:13.065 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.065 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.065 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.065 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.066 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6803", 2021-07-26T06:13:13.066 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.066 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.067 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.067 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.068 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:13:13.068 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.069 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.069 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.070 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6804", 2021-07-26T06:13:13.070 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.070 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.071 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.071 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.071 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6805", 2021-07-26T06:13:13.071 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.071 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.072 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.072 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.072 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:13:13.072 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.073 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.073 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.073 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6808", 2021-07-26T06:13:13.073 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.073 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.074 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.074 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.075 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6809", 2021-07-26T06:13:13.075 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.076 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.076 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.076 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.076 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:13:13.077 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.077 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.077 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.078 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6806", 2021-07-26T06:13:13.078 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.078 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.078 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.079 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.079 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6807", 2021-07-26T06:13:13.079 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:13.079 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.079 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.080 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.080 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6803/52124", 2021-07-26T06:13:13.080 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6805/52124", 2021-07-26T06:13:13.080 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6809/52124", 2021-07-26T06:13:13.081 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6807/52124", 2021-07-26T06:13:13.081 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:13:13.081 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:13:13.081 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:13:13.081 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.082 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.082 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.082 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:13.082 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "64b2a2e5-8ab4-4197-9d59-4b39f363ac4c", 2021-07-26T06:13:13.083 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 0, 2021-07-26T06:13:13.083 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:13:13.083 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:13:13.083 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:13:13.084 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 0, 2021-07-26T06:13:13.084 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 0, 2021-07-26T06:13:13.084 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 10, 2021-07-26T06:13:13.084 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 35, 2021-07-26T06:13:13.084 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 39, 2021-07-26T06:13:13.085 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:13:13.085 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:13.085 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.085 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.086 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.086 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:13:13.086 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.086 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.086 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.087 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.087 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:13:13.087 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.088 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.088 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.089 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.089 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:13:13.089 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.089 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.090 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.090 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:13:13.090 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.090 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.091 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.091 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.091 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:13:13.091 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.092 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.092 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.092 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.092 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:13:13.092 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.093 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.093 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.093 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:13:13.093 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.093 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.094 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.094 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.094 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:13:13.094 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.095 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.095 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.095 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.095 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:13:13.095 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.096 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.096 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.096 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:13:13.097 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.097 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.097 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.097 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.098 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:13:13.098 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:13.098 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.098 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.099 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.099 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/46333", 2021-07-26T06:13:13.100 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/46333", 2021-07-26T06:13:13.100 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/46333", 2021-07-26T06:13:13.100 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/46333", 2021-07-26T06:13:13.100 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:13:13.101 INFO:tasks.workunit.client.0.smithi094.stdout: "exists" 2021-07-26T06:13:13.101 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.101 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.101 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.102 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:13.102 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "8f418200-be0a-4ee5-8381-ae936884bd8d", 2021-07-26T06:13:13.102 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:13:13.102 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:13:13.102 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:13:13.103 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:13:13.103 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 15, 2021-07-26T06:13:13.103 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 37, 2021-07-26T06:13:13.103 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 46, 2021-07-26T06:13:13.104 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 52, 2021-07-26T06:13:13.104 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 40, 2021-07-26T06:13:13.104 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:13:13.104 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:13.104 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.105 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.105 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.105 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:13:13.105 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.105 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.106 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.106 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.106 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:13:13.106 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.107 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.107 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.107 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.107 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:13:13.107 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.108 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.108 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.108 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:13:13.108 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.109 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.109 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.109 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.109 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:13:13.109 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.110 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.110 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.110 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.110 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:13:13.110 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.111 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.111 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.111 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:13:13.111 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.112 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.112 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.112 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.112 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:13:13.112 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.113 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.113 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.113 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.113 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:13:13.114 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:13.114 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.114 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:13.115 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:13:13.115 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.115 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.115 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.116 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:13.116 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:13:13.116 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:13.116 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.117 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.117 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.117 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/52704", 2021-07-26T06:13:13.117 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/52704", 2021-07-26T06:13:13.117 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/52704", 2021-07-26T06:13:13.118 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/52704", 2021-07-26T06:13:13.118 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:13:13.118 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:13:13.118 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:13:13.118 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.119 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.119 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.119 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_xinfo": [ 2021-07-26T06:13:13.119 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.120 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:13.120 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:12:21.330001+0000", 2021-07-26T06:13:13.120 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:13:13.120 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:13:13.120 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:13:13.121 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:13:13.121 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:11:00.710074+0000", 2021-07-26T06:13:13.121 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:13:13.121 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.121 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.122 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:13.122 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:12:21.858488+0000", 2021-07-26T06:13:13.122 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:13:13.122 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:13:13.123 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:13:13.123 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:13:13.123 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:11:12.450854+0000", 2021-07-26T06:13:13.123 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:13:13.124 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.124 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.124 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:13.124 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:12:22.519390+0000", 2021-07-26T06:13:13.125 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:13:13.125 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:13:13.125 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:13:13.125 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:13:13.125 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:11:25.473164+0000", 2021-07-26T06:13:13.126 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:13:13.126 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.126 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.126 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap": [], 2021-07-26T06:13:13.127 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap_items": [], 2021-07-26T06:13:13.127 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_temp": [], 2021-07-26T06:13:13.127 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_temp": [], 2021-07-26T06:13:13.127 INFO:tasks.workunit.client.0.smithi094.stdout: "blocklist": {}, 2021-07-26T06:13:13.127 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profiles": { 2021-07-26T06:13:13.128 INFO:tasks.workunit.client.0.smithi094.stdout: "default": { 2021-07-26T06:13:13.128 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:13:13.128 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "2", 2021-07-26T06:13:13.128 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:13:13.128 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van" 2021-07-26T06:13:13.129 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.129 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.129 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps_queue": [], 2021-07-26T06:13:13.129 INFO:tasks.workunit.client.0.smithi094.stdout: "new_removed_snaps": [], 2021-07-26T06:13:13.130 INFO:tasks.workunit.client.0.smithi094.stdout: "new_purged_snaps": [], 2021-07-26T06:13:13.130 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_node_flags": {}, 2021-07-26T06:13:13.130 INFO:tasks.workunit.client.0.smithi094.stdout: "device_class_flags": {}, 2021-07-26T06:13:13.130 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": { 2021-07-26T06:13:13.130 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_enabled": false, 2021-07-26T06:13:13.131 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_bucket_count": 0, 2021-07-26T06:13:13.131 INFO:tasks.workunit.client.0.smithi094.stdout: "degraded_stretch_mode": 0, 2021-07-26T06:13:13.131 INFO:tasks.workunit.client.0.smithi094.stdout: "recovering_stretch_mode": 0, 2021-07-26T06:13:13.131 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_bucket": 0 2021-07-26T06:13:13.131 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.132 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.132 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_metadata": [ 2021-07-26T06:13:13.132 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.132 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:13.133 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:13:13.133 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6804/52124,v1:127.0.0.1:6805/52124]", 2021-07-26T06:13:13.133 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:13:13.133 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:13:13.133 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:13:13.134 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:13:13.134 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:13:13.134 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:13:13.134 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:13:13.134 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:13:13.135 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:13:13.135 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-07-26T06:13:13.135 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:13:13.135 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:13:13.136 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:13:13.136 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:13:13.136 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:13:13.136 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:13:13.136 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:13:13.137 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:13:13.137 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:13:13.137 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:13:13.137 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:13:13.138 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:13:13.138 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:13:13.138 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:13:13.138 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:13:13.138 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6802/52124,v1:127.0.0.1:6803/52124]", 2021-07-26T06:13:13.139 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:13:13.139 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/52124,v1:127.0.0.1:6809/52124]", 2021-07-26T06:13:13.139 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/52124,v1:127.0.0.1:6807/52124]", 2021-07-26T06:13:13.139 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:13:13.140 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:13:13.140 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:13:13.140 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:13:13.140 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:13:13.140 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:13:13.141 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:13:13.141 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:13:13.141 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:13:13.141 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/0", 2021-07-26T06:13:13.142 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:13:13.142 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:13:13.142 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:13:13.142 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.142 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.143 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:13.143 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:13:13.143 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/46333,v1:127.0.0.1:6813/46333]", 2021-07-26T06:13:13.143 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:13:13.143 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:13:13.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:13:13.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:13:13.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:13:13.144 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:13:13.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:13:13.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:13:13.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:13:13.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-07-26T06:13:13.145 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:13:13.146 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:13:13.146 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:13:13.146 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:13:13.146 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:13:13.147 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:13:13.147 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:13:13.147 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:13:13.147 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:13:13.147 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:13:13.148 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:13:13.148 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:13:13.148 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:13:13.148 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:13:13.148 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:13:13.149 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/46333,v1:127.0.0.1:6811/46333]", 2021-07-26T06:13:13.149 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:13:13.149 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/46333,v1:127.0.0.1:6817/46333]", 2021-07-26T06:13:13.149 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/46333,v1:127.0.0.1:6815/46333]", 2021-07-26T06:13:13.150 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:13:13.150 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:13:13.150 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:13:13.150 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:13:13.150 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:13:13.151 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:13:13.151 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:13:13.151 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:13:13.151 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:13:13.151 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/1", 2021-07-26T06:13:13.152 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:13:13.152 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:13:13.152 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:13:13.152 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.153 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.153 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:13.153 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:13:13.153 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/52704,v1:127.0.0.1:6813/52704]", 2021-07-26T06:13:13.153 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:13:13.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:13:13.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:13:13.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:13:13.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:13:13.154 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:13:13.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:13:13.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:13:13.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:13:13.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-07-26T06:13:13.155 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:13:13.156 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:13:13.156 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:13:13.156 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:13:13.156 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:13:13.157 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:13:13.157 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:13:13.157 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:13:13.157 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:13:13.157 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:13:13.158 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:13:13.158 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:13:13.158 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:13:13.158 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:13:13.159 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:13:13.159 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/52704,v1:127.0.0.1:6811/52704]", 2021-07-26T06:13:13.159 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:13:13.159 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/52704,v1:127.0.0.1:6817/52704]", 2021-07-26T06:13:13.159 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/52704,v1:127.0.0.1:6815/52704]", 2021-07-26T06:13:13.160 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:13:13.160 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:13:13.160 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:13:13.161 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:13:13.161 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:13:13.161 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:13:13.162 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:13:13.162 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:13:13.162 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:13:13.163 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/2", 2021-07-26T06:13:13.163 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:13:13.164 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:13:13.164 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:13:13.164 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.165 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.165 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_clean_epochs": { 2021-07-26T06:13:13.165 INFO:tasks.workunit.client.0.smithi094.stdout: "min_last_epoch_clean": 0, 2021-07-26T06:13:13.166 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": { 2021-07-26T06:13:13.166 INFO:tasks.workunit.client.0.smithi094.stdout: "per_pool": [ 2021-07-26T06:13:13.166 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.167 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:13:13.167 INFO:tasks.workunit.client.0.smithi094.stdout: "floor": 36 2021-07-26T06:13:13.167 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.167 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.167 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.168 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_epochs": [ 2021-07-26T06:13:13.168 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.168 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:13.168 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 44 2021-07-26T06:13:13.169 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.169 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.169 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:13.169 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 10 2021-07-26T06:13:13.170 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.170 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.170 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:13.170 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 46 2021-07-26T06:13:13.170 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.171 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.171 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.171 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_first_committed": 1, 2021-07-26T06:13:13.171 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_last_committed": 53, 2021-07-26T06:13:13.172 INFO:tasks.workunit.client.0.smithi094.stdout: "crushmap": { 2021-07-26T06:13:13.172 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": [ 2021-07-26T06:13:13.172 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.172 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:13.173 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.0", 2021-07-26T06:13:13.173 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:13:13.173 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.173 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.174 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:13.174 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.1", 2021-07-26T06:13:13.174 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:13:13.174 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.174 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.175 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:13.175 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.2", 2021-07-26T06:13:13.175 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:13:13.176 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.176 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.176 INFO:tasks.workunit.client.0.smithi094.stdout: "types": [ 2021-07-26T06:13:13.176 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.176 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 0, 2021-07-26T06:13:13.177 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd" 2021-07-26T06:13:13.177 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.177 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.178 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:13:13.178 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "host" 2021-07-26T06:13:13.178 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.178 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.179 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 2, 2021-07-26T06:13:13.179 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "chassis" 2021-07-26T06:13:13.179 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.180 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.180 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 3, 2021-07-26T06:13:13.180 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "rack" 2021-07-26T06:13:13.180 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.181 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.181 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 4, 2021-07-26T06:13:13.181 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "row" 2021-07-26T06:13:13.182 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.182 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.182 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 5, 2021-07-26T06:13:13.182 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pdu" 2021-07-26T06:13:13.183 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.183 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.183 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 6, 2021-07-26T06:13:13.183 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pod" 2021-07-26T06:13:13.184 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.184 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.184 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 7, 2021-07-26T06:13:13.184 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "room" 2021-07-26T06:13:13.185 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.185 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.185 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 8, 2021-07-26T06:13:13.185 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "datacenter" 2021-07-26T06:13:13.186 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.186 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.186 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 9, 2021-07-26T06:13:13.186 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "zone" 2021-07-26T06:13:13.187 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.187 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.187 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 10, 2021-07-26T06:13:13.188 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "region" 2021-07-26T06:13:13.188 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.188 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.188 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:13:13.189 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "root" 2021-07-26T06:13:13.189 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.189 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.189 INFO:tasks.workunit.client.0.smithi094.stdout: "buckets": [ 2021-07-26T06:13:13.190 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.190 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -1, 2021-07-26T06:13:13.190 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default", 2021-07-26T06:13:13.190 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:13:13.191 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:13:13.191 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:13.191 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:13.191 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:13.192 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:13.192 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.192 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:13:13.192 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:13.193 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:13.193 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.193 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.193 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.194 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.194 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -2, 2021-07-26T06:13:13.194 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default~hdd", 2021-07-26T06:13:13.195 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:13:13.195 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:13:13.195 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:13.195 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:13.196 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:13.196 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:13.196 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.196 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:13:13.197 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:13.197 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:13.197 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.198 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.198 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.198 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.198 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:13:13.199 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094", 2021-07-26T06:13:13.199 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:13:13.199 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:13:13.199 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:13.200 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:13.200 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:13.200 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:13.200 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.201 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:13.201 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:13.201 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:13.201 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.202 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.202 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:13.202 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:13.202 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:13:13.203 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.203 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.203 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:13.203 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:13.204 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:13:13.204 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.204 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.204 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.205 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.205 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:13:13.205 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094~hdd", 2021-07-26T06:13:13.205 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:13:13.205 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:13:13.206 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:13.206 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:13.206 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:13.206 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:13.207 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.207 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:13.207 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:13.207 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:13.207 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.208 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.208 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:13.208 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:13.208 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:13:13.209 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.209 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.209 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:13.209 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:13.209 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:13:13.210 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.210 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.210 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.211 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.211 INFO:tasks.workunit.client.0.smithi094.stdout: "rules": [ 2021-07-26T06:13:13.211 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.211 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 0, 2021-07-26T06:13:13.212 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "replicated_rule", 2021-07-26T06:13:13.212 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:13:13.212 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:13:13.212 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.213 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:13:13.213 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:13:13.213 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:13:13.213 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.214 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.214 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_firstn", 2021-07-26T06:13:13.214 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:13:13.214 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:13:13.214 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.215 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.215 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:13:13.215 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.216 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:13.216 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.216 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.217 INFO:tasks.workunit.client.0.smithi094.stdout: "tunables": { 2021-07-26T06:13:13.217 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_tries": 0, 2021-07-26T06:13:13.217 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_fallback_tries": 0, 2021-07-26T06:13:13.218 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_total_tries": 50, 2021-07-26T06:13:13.218 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_descend_once": 1, 2021-07-26T06:13:13.218 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_vary_r": 1, 2021-07-26T06:13:13.218 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_stable": 1, 2021-07-26T06:13:13.219 INFO:tasks.workunit.client.0.smithi094.stdout: "straw_calc_version": 1, 2021-07-26T06:13:13.219 INFO:tasks.workunit.client.0.smithi094.stdout: "allowed_bucket_algs": 54, 2021-07-26T06:13:13.220 INFO:tasks.workunit.client.0.smithi094.stdout: "profile": "jewel", 2021-07-26T06:13:13.220 INFO:tasks.workunit.client.0.smithi094.stdout: "optimal_tunables": 1, 2021-07-26T06:13:13.220 INFO:tasks.workunit.client.0.smithi094.stdout: "legacy_tunables": 0, 2021-07-26T06:13:13.220 INFO:tasks.workunit.client.0.smithi094.stdout: "minimum_required_version": "jewel", 2021-07-26T06:13:13.221 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables": 1, 2021-07-26T06:13:13.221 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables2": 1, 2021-07-26T06:13:13.222 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v2_rules": 0, 2021-07-26T06:13:13.222 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables3": 1, 2021-07-26T06:13:13.222 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v3_rules": 0, 2021-07-26T06:13:13.222 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v4_buckets": 1, 2021-07-26T06:13:13.223 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables5": 1, 2021-07-26T06:13:13.223 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v5_rules": 0 2021-07-26T06:13:13.223 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.224 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_args": {} 2021-07-26T06:13:13.224 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.224 INFO:tasks.workunit.client.0.smithi094.stdout: "fsmap": { 2021-07-26T06:13:13.225 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:13:13.225 INFO:tasks.workunit.client.0.smithi094.stdout: "default_fscid": -1, 2021-07-26T06:13:13.225 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": { 2021-07-26T06:13:13.225 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": {}, 2021-07-26T06:13:13.226 INFO:tasks.workunit.client.0.smithi094.stdout: "ro_compat": {}, 2021-07-26T06:13:13.226 INFO:tasks.workunit.client.0.smithi094.stdout: "incompat": { 2021-07-26T06:13:13.226 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_1": "base v0.20", 2021-07-26T06:13:13.226 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_2": "client writeable ranges", 2021-07-26T06:13:13.227 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_3": "default file layouts on dirs", 2021-07-26T06:13:13.227 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_4": "dir inode in separate object", 2021-07-26T06:13:13.227 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_5": "mds uses versioned encoding", 2021-07-26T06:13:13.227 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_6": "dirfrag is stored in omap", 2021-07-26T06:13:13.227 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_8": "no anchor table", 2021-07-26T06:13:13.228 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_9": "file layout v2", 2021-07-26T06:13:13.228 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_10": "snaprealm v2" 2021-07-26T06:13:13.228 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.228 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.229 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_flags": { 2021-07-26T06:13:13.229 INFO:tasks.workunit.client.0.smithi094.stdout: "enable_multiple": true, 2021-07-26T06:13:13.229 INFO:tasks.workunit.client.0.smithi094.stdout: "ever_enabled_multiple": true 2021-07-26T06:13:13.229 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.229 INFO:tasks.workunit.client.0.smithi094.stdout: "standbys": [], 2021-07-26T06:13:13.230 INFO:tasks.workunit.client.0.smithi094.stdout: "filesystems": [] 2021-07-26T06:13:13.230 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.230 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_first_committed": 1, 2021-07-26T06:13:13.230 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_last_committed": 1, 2021-07-26T06:13:13.231 INFO:tasks.workunit.client.0.smithi094.stdout: "auth": { 2021-07-26T06:13:13.231 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:13:13.231 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 5, 2021-07-26T06:13:13.231 INFO:tasks.workunit.client.0.smithi094.stdout: "num_secrets": 3 2021-07-26T06:13:13.231 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 128, 2021-07-26T06:13:13.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_active": 128, 2021-07-26T06:13:13.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_unknown": 0, 2021-07-26T06:13:13.232 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osd": 3, 2021-07-26T06:13:13.232 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_sum": { 2021-07-26T06:13:13.233 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:13:13.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61878, 2021-07-26T06:13:13.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 99, 2021-07-26T06:13:13.233 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:13:13.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 297, 2021-07-26T06:13:13.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:13:13.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:13:13.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 99, 2021-07-26T06:13:13.234 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:13:13.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:13:13.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 99, 2021-07-26T06:13:13.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:13:13.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:13:13.235 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:13:13.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 94, 2021-07-26T06:13:13.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 94, 2021-07-26T06:13:13.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:13:13.236 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:13:13.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:13:13.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:13:13.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:13:13.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:13:13.237 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:13:13.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:13:13.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:13:13.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:13:13.238 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:13:13.239 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:13:13.239 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:13:13.239 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:13:13.239 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:13:13.240 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:13:13.240 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:13:13.240 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:13:13.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:13:13.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:13:13.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:13:13.241 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:13:13.242 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:13:13.242 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:13:13.242 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:13:13.242 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.243 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:13:13.243 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:13:13.243 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:13:13.244 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:13.244 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 0, 2021-07-26T06:13:13.244 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 0, 2021-07-26T06:13:13.244 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:13.245 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:13.245 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:13.245 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:13:13.245 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:13:13.245 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.246 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 2181, 2021-07-26T06:13:13.246 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 2181, 2021-07-26T06:13:13.246 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 256, 2021-07-26T06:13:13.246 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 256, 2021-07-26T06:13:13.246 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 0 2021-07-26T06:13:13.247 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.247 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum": { 2021-07-26T06:13:13.247 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:13:13.247 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:13:13.247 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 384, 2021-07-26T06:13:13.248 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:13:13.248 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:13:13.248 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:13:13.248 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:13:13.249 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 24552, 2021-07-26T06:13:13.249 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2384, 2021-07-26T06:13:13.249 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:13:13.249 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 22141, 2021-07-26T06:13:13.249 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314548248, 2021-07-26T06:13:13.250 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:13:13.250 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:13:13.250 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322097405952, 2021-07-26T06:13:13.250 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:13.251 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2441216, 2021-07-26T06:13:13.251 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 589757, 2021-07-26T06:13:13.251 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:13.251 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:13.252 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:13.252 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:13:13.252 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 22673296 2021-07-26T06:13:13.252 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.252 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:13:13.253 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:13:13.253 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:13:13.253 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:13:13.253 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:13:13.253 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:13:13.254 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:13:13.254 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.254 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:13:13.254 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 50, 2021-07-26T06:13:13.255 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 50, 2021-07-26T06:13:13.255 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 50000000, 2021-07-26T06:13:13.255 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 50000000 2021-07-26T06:13:13.255 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.255 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:13:13.256 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:13:13.256 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.256 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum_by_class": { 2021-07-26T06:13:13.256 INFO:tasks.workunit.client.0.smithi094.stdout: "hdd": { 2021-07-26T06:13:13.256 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:13:13.257 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:13:13.257 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 384, 2021-07-26T06:13:13.257 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:13:13.257 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:13:13.258 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:13:13.258 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:13:13.258 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 24552, 2021-07-26T06:13:13.258 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2384, 2021-07-26T06:13:13.258 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:13:13.259 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 22141, 2021-07-26T06:13:13.259 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314548248, 2021-07-26T06:13:13.259 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:13:13.259 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:13:13.259 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322097405952, 2021-07-26T06:13:13.260 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:13.260 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2441216, 2021-07-26T06:13:13.260 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 589757, 2021-07-26T06:13:13.260 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:13.261 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:13.261 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:13.261 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:13:13.261 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 22673296 2021-07-26T06:13:13.261 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.262 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:13:13.262 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:13:13.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:13:13.262 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:13:13.262 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:13:13.263 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:13:13.263 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:13:13.263 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.263 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:13:13.264 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 50, 2021-07-26T06:13:13.264 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 50, 2021-07-26T06:13:13.264 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 50000000, 2021-07-26T06:13:13.264 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 50000000 2021-07-26T06:13:13.264 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.265 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:13:13.265 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:13:13.265 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.265 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.266 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_stats": [ 2021-07-26T06:13:13.266 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.266 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:13:13.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 128, 2021-07-26T06:13:13.267 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:13:13.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 61878, 2021-07-26T06:13:13.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 99, 2021-07-26T06:13:13.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:13:13.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 297, 2021-07-26T06:13:13.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:13:13.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:13:13.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 99, 2021-07-26T06:13:13.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:13:13.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:13:13.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 99, 2021-07-26T06:13:13.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:13:13.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:13:13.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:13:13.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 94, 2021-07-26T06:13:13.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 94, 2021-07-26T06:13:13.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:13:13.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:13:13.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:13:13.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:13:13.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:13:13.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:13:13.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:13:13.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:13:13.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:13:13.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:13:13.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:13:13.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:13:13.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:13:13.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:13:13.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:13:13.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:13:13.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:13:13.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:13:13.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:13:13.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:13:13.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:13:13.275 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:13:13.275 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:13:13.275 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:13:13.275 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:13:13.275 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.276 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:13:13.276 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:13:13.276 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:13:13.276 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:13.277 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 1220608, 2021-07-26T06:13:13.277 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 184152, 2021-07-26T06:13:13.277 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:13.277 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:13.277 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:13.278 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:13:13.278 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:13:13.278 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.278 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 2181, 2021-07-26T06:13:13.278 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 2181, 2021-07-26T06:13:13.279 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 256, 2021-07-26T06:13:13.279 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 256, 2021-07-26T06:13:13.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 3 2021-07-26T06:13:13.279 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.279 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.280 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_stats": [ 2021-07-26T06:13:13.280 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.280 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:13.280 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 188978561035 2021-07-26T06:13:13.281 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.281 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.281 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:13.281 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 42949672981 2021-07-26T06:13:13.282 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.282 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.282 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:13.282 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 197568495626 2021-07-26T06:13:13.283 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.283 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.283 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_state": [ 2021-07-26T06:13:13.284 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.284 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized+degraded", 2021-07-26T06:13:13.285 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 93 2021-07-26T06:13:13.285 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.285 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.286 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized", 2021-07-26T06:13:13.286 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 35 2021-07-26T06:13:13.286 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.286 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.287 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_osd": [ 2021-07-26T06:13:13.287 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.287 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:13.287 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 59, 2021-07-26T06:13:13.287 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 128, 2021-07-26T06:13:13.288 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:13:13.288 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.288 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.288 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:13.289 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 69, 2021-07-26T06:13:13.289 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 128, 2021-07-26T06:13:13.289 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:13:13.289 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.290 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.290 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:13.290 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:13:13.291 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 0, 2021-07-26T06:13:13.291 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:13:13.291 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.292 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.292 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": [ 2021-07-26T06:13:13.292 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:13.292 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:13:13.293 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": {} 2021-07-26T06:13:13.293 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.293 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:13.293 INFO:tasks.workunit.client.0.smithi094.stdout: "servicemap": { 2021-07-26T06:13:13.293 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 5, 2021-07-26T06:13:13.294 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:12:27.832513+0000", 2021-07-26T06:13:13.294 INFO:tasks.workunit.client.0.smithi094.stdout: "services": {} 2021-07-26T06:13:13.294 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:13.294 INFO:tasks.workunit.client.0.smithi094.stdout: "paxos": { 2021-07-26T06:13:13.295 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:13:13.295 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 448, 2021-07-26T06:13:13.295 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pn": 0, 2021-07-26T06:13:13.295 INFO:tasks.workunit.client.0.smithi094.stdout: "accepted_pn": 0 2021-07-26T06:13:13.295 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:13.296 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:13:13.297 INFO:tasks.workunit.client.0.smithi094.stderr:report 2921000546 2021-07-26T06:13:13.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: return 1 2021-07-26T06:13:13.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2021-07-26T06:13:13.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2021-07-26T06:13:13.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2021-07-26T06:13:13.298 INFO:tasks.workunit.client.0.smithi094.stdout:writing non-divergent object existing_4 2021-07-26T06:13:13.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:13:13.342 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:13:13.342 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.656870+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.343 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061855+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.343 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.056238+0000 34'96 52:135 [0,2] 0 [0,2] 0 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.343 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059226+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.343 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 1 0 0 617 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651961+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.343 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055365+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.344 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 1 0 0 617 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650564+0000 34'96 53:145 [2,0] 2 [2,0] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.344 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 3 0 0 1851 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651681+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.344 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161682+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.344 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056354+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.345 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063687+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.345 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060493+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.345 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647254+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.345 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649803+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.346 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650413+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.346 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064477+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.346 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.648792+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:13:13.346 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649585+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.346 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061742+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.347 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648454+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.347 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649644+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.347 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645510+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.347 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:36.981078+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.348 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054380+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.348 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645644+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.348 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059874+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.348 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.656499+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.348 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061665+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.349 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.165344+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.349 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057712+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.349 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160614+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.349 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.655228+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.349 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652396+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652639+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647082+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161648+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644788+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161366+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650536+0000 34'96 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057570+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062423+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061857+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651257+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.650322+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064099+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060925+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646360+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.655464+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648831+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059519+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055783+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058878+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055308+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646801+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057260+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059379+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.056265+0000 32'95 52:138 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649448+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055108+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646439+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056459+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.059557+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058386+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060541+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651884+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063303+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060893+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652020+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063243+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651093+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644530+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061408+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061302+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159501+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160463+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.654813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063624+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652317+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062782+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651456+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159722+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648810+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652809+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651444+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054311+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160718+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650851+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058230+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651009+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.163179+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.592753+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064015+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647935+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064861+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.159494+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652699+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.159523+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.062968+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647259+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.162249+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.056263+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645544+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.054889+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.161101+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644646+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.656444+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.058220+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.059923+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645471+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652814+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652308+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644586+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648452+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.646610+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.057203+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648029+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655913+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.060941+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655011+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.645795+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.054246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058839+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.162128+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058635+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.371 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060058+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:13.371 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:13:13.371 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:13:13.372 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:13:13.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2021-07-26T06:13:13.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:13:13.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2021-07-26T06:13:13.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:13:13.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:13:13.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:13:13.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:13:13.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:13:13.512 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:13:13.512 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:13:13.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:13:13.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:13:13.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:13:13.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:13:13.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:13:13.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:13:13.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:13:13.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:13:14.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:13:14.030 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:13:14.030 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:13:14.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:13:14.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:14.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:13:14.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=188978561038 2021-07-26T06:13:14.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 188978561038 2021-07-26T06:13:14.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-188978561038' 2021-07-26T06:13:14.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:14.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:13:14.340 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-07-26T06:13:14.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T06:13:14.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T06:13:14.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T06:13:14.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:13:14.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:13:14.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495629 2021-07-26T06:13:14.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495629 2021-07-26T06:13:14.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-188978561038 2-197568495629' 2021-07-26T06:13:14.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:14.491 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-188978561038 2021-07-26T06:13:14.491 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:14.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:13:14.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-188978561038 2021-07-26T06:13:14.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:14.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=188978561038 2021-07-26T06:13:14.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 188978561038' 2021-07-26T06:13:14.495 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 188978561038 2021-07-26T06:13:14.495 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:13:14.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 188978561036 -lt 188978561038 2021-07-26T06:13:14.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:13:15.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:13:15.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:13:16.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 188978561038 -lt 188978561038 2021-07-26T06:13:16.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:13:16.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-197568495629 2021-07-26T06:13:16.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:13:16.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:13:16.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-197568495629 2021-07-26T06:13:16.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:13:16.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495629 2021-07-26T06:13:16.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 197568495629' 2021-07-26T06:13:16.241 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 197568495629 2021-07-26T06:13:16.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:13:16.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495629 -lt 197568495629 2021-07-26T06:13:16.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:13:16.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:16.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 128 == 0 2021-07-26T06:13:17.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:17.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:17.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:17.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:17.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:17.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:17.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:17.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:17.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:17.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:17.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:18.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:18.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:13:18.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:13:18.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:13:18.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:18.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:13:18.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:18.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:18.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:18.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:18.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:18.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:18.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:18.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:18.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:18.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:18.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:18.535 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:19.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:19.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:19.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:19.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:19.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:19.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:19.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:19.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:19.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:19.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:19.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:19.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:13:19.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:19.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:13:19.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:19.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:19.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:19.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:19.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:19.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:19.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:19.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:20.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:20.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:20.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:20.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:20.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:20.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:20.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:20.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:20.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:20.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:20.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:20.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:20.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:21.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:21.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:21.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:13:21.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:21.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:13:21.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:21.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:21.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:21.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:21.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:21.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:21.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:21.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:21.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:21.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:21.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:21.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:22.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:22.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:22.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:22.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:22.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:22.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:22.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:22.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:22.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:22.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:22.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:22.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:13:22.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:22.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:13:23.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:23.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:23.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:23.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:23.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:23.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:23.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:23.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:24.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:24.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:24.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:24.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:24.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:24.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:24.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:24.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:24.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:24.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:24.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:24.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:24.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:25.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:25.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:25.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 4 >= 8 )) 2021-07-26T06:13:25.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:25.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 1.6 2021-07-26T06:13:26.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:26.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:26.702 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:26.702 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:26.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:26.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:26.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:26.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:27.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:27.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:27.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:27.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:27.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:27.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:27.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:27.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:27.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:27.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:27.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:27.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:27.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:28.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:28.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:28.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 5 >= 8 )) 2021-07-26T06:13:28.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:28.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 3.2 2021-07-26T06:13:31.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:31.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:31.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:31.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:31.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:31.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:31.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:31.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:31.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:31.697 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:31.697 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:31.698 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:32.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:32.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:32.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:32.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:32.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:32.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:32.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:32.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:32.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:32.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:32.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:32.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 6 >= 8 )) 2021-07-26T06:13:32.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:32.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 6.4 2021-07-26T06:13:39.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:39.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:39.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:39.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:39.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:39.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:39.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:39.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:39.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:39.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:39.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:39.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:40.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:40.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:40.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:40.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:40.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:40.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:40.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:40.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:40.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:40.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:40.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:40.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 7 >= 8 )) 2021-07-26T06:13:40.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:13:40.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 7.3 2021-07-26T06:13:47.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:13:47.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:13:47.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:13:47.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:13:47.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:13:47.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:13:47.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:13:47.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:13:48.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:13:48.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:13:48.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:13:48.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:13:48.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 128 2021-07-26T06:13:48.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:13:48.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:13:48.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:13:48.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:13:48.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:13:48.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:13:48.757 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:13:48.757 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:13:49.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:13:49.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:13:49.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 8 >= 8 )) 2021-07-26T06:13:49.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph report 2021-07-26T06:13:49.661 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:13:49.661 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_fingerprint": "2e823565-04be-471a-8f59-734ee1c21e89", 2021-07-26T06:13:49.662 INFO:tasks.workunit.client.0.smithi094.stdout: "version": "17.0.0-6358-g197e564f", 2021-07-26T06:13:49.662 INFO:tasks.workunit.client.0.smithi094.stdout: "commit": "197e564fdaee12596de1dc8f3253d1704ae66d5d", 2021-07-26T06:13:49.662 INFO:tasks.workunit.client.0.smithi094.stdout: "timestamp": "2021-07-26T06:13:49.659554+0000", 2021-07-26T06:13:49.662 INFO:tasks.workunit.client.0.smithi094.stdout: "tag": "", 2021-07-26T06:13:49.662 INFO:tasks.workunit.client.0.smithi094.stdout: "health": { 2021-07-26T06:13:49.663 INFO:tasks.workunit.client.0.smithi094.stdout: "status": "HEALTH_WARN", 2021-07-26T06:13:49.663 INFO:tasks.workunit.client.0.smithi094.stdout: "checks": { 2021-07-26T06:13:49.663 INFO:tasks.workunit.client.0.smithi094.stdout: "OSDMAP_FLAGS": { 2021-07-26T06:13:49.663 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:49.664 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:49.664 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "nodown,noout,noin flag(s) set", 2021-07-26T06:13:49.664 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 17 2021-07-26T06:13:49.664 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.665 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [], 2021-07-26T06:13:49.665 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:49.665 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.665 INFO:tasks.workunit.client.0.smithi094.stdout: "OSD_DOWN": { 2021-07-26T06:13:49.665 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:49.666 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:49.666 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 osds down", 2021-07-26T06:13:49.666 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:13:49.666 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.667 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:13:49.667 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.667 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "osd.1 (root=default,host=smithi094) is down" 2021-07-26T06:13:49.667 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.667 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.668 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:49.668 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.668 INFO:tasks.workunit.client.0.smithi094.stdout: "PG_DEGRADED": { 2021-07-26T06:13:49.668 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:49.669 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:49.669 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "Degraded data redundancy: 99/297 objects degraded (33.333%), 93 pgs degraded, 128 pgs undersized", 2021-07-26T06:13:49.669 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 221 2021-07-26T06:13:49.669 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.669 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:13:49.670 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.670 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.0 is stuck undersized for 71s, current state active+undersized, last acting [0,2]" 2021-07-26T06:13:49.670 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.670 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.671 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.671 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.671 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.671 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.671 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.672 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.672 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.3 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.672 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.672 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.673 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.4 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.673 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.673 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.673 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.5 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.673 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.674 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.674 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.6 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.674 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.674 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.675 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.7 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.675 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.675 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.675 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.8 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.675 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.676 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.676 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.9 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.676 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.676 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.676 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.a is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.677 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.677 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.677 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.b is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.677 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.678 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.678 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.c is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.678 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.678 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.678 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.d is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.679 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.679 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.679 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.e is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.679 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.679 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.680 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.f is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.680 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.680 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.680 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.10 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.680 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.681 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.681 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2021-07-26T06:13:49.681 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.681 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.681 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1c is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.682 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.682 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.682 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1d is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.682 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.683 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.683 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1e is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.683 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.684 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.684 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.1f is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.684 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.684 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.685 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.20 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.685 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.685 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.685 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.21 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.685 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.686 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.686 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.22 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.686 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.686 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.687 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.23 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.687 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.687 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.687 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.24 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.687 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.688 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.688 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.25 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.688 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.688 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.688 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.26 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.689 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.689 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.689 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.27 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.689 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.689 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.690 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.28 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.690 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.690 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.690 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.29 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.690 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.691 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.691 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2a is stuck undersized for 70s, current state active+undersized, last acting [2,0]" 2021-07-26T06:13:49.691 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.691 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.691 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2b is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.692 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.692 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.692 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2c is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.692 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.692 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.693 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2d is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.693 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.693 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.693 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2e is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.694 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.694 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.694 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.2f is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.694 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.694 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.695 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.30 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.695 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.695 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.695 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.31 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.696 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.696 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.696 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.32 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.696 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.697 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.697 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.33 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.697 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.697 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.697 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.34 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.698 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.698 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.698 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.35 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.698 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.699 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.699 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.36 is stuck undersized for 70s, current state active+undersized+degraded, last acting [2,0]" 2021-07-26T06:13:49.699 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.699 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.699 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.37 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.700 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.700 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.700 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.38 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.700 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.700 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.700 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.39 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.701 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.701 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.701 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.3a is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-07-26T06:13:49.701 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.702 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.702 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.7e is stuck undersized for 70s, current state active+undersized, last acting [2,0]" 2021-07-26T06:13:49.702 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.702 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.702 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.7f is stuck undersized for 70s, current state active+undersized, last acting [2,0]" 2021-07-26T06:13:49.703 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.703 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.703 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:49.703 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.703 INFO:tasks.workunit.client.0.smithi094.stdout: "POOL_APP_NOT_ENABLED": { 2021-07-26T06:13:49.704 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:13:49.704 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:13:49.704 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 pool(s) do not have an application enabled", 2021-07-26T06:13:49.704 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:13:49.704 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.705 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:13:49.705 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.705 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "application not enabled on pool 'test'" 2021-07-26T06:13:49.705 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.705 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.706 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-07-26T06:13:49.706 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.706 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.706 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:13:49.706 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.707 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.707 INFO:tasks.workunit.client.0.smithi094.stdout: "mutes": [] 2021-07-26T06:13:49.707 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.707 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_first_committed": 1, 2021-07-26T06:13:49.707 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_last_committed": 1, 2021-07-26T06:13:49.708 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap": { 2021-07-26T06:13:49.708 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:13:49.708 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:13:49.708 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:10:48.128736Z", 2021-07-26T06:13:49.708 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:10:48.128736Z", 2021-07-26T06:13:49.709 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release": 17, 2021-07-26T06:13:49.709 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release_name": "quincy", 2021-07-26T06:13:49.709 INFO:tasks.workunit.client.0.smithi094.stdout: "election_strategy": 1, 2021-07-26T06:13:49.709 INFO:tasks.workunit.client.0.smithi094.stdout: "disallowed_leaders: ": "", 2021-07-26T06:13:49.709 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": false, 2021-07-26T06:13:49.710 INFO:tasks.workunit.client.0.smithi094.stdout: "features": { 2021-07-26T06:13:49.710 INFO:tasks.workunit.client.0.smithi094.stdout: "persistent": [ 2021-07-26T06:13:49.710 INFO:tasks.workunit.client.0.smithi094.stdout: "kraken", 2021-07-26T06:13:49.710 INFO:tasks.workunit.client.0.smithi094.stdout: "luminous", 2021-07-26T06:13:49.710 INFO:tasks.workunit.client.0.smithi094.stdout: "mimic", 2021-07-26T06:13:49.711 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap-prune", 2021-07-26T06:13:49.711 INFO:tasks.workunit.client.0.smithi094.stdout: "nautilus", 2021-07-26T06:13:49.711 INFO:tasks.workunit.client.0.smithi094.stdout: "octopus", 2021-07-26T06:13:49.711 INFO:tasks.workunit.client.0.smithi094.stdout: "pacific", 2021-07-26T06:13:49.711 INFO:tasks.workunit.client.0.smithi094.stdout: "elector-pinging", 2021-07-26T06:13:49.712 INFO:tasks.workunit.client.0.smithi094.stdout: "quincy" 2021-07-26T06:13:49.712 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.712 INFO:tasks.workunit.client.0.smithi094.stdout: "optional": [] 2021-07-26T06:13:49.712 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.712 INFO:tasks.workunit.client.0.smithi094.stdout: "mons": [ 2021-07-26T06:13:49.713 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.713 INFO:tasks.workunit.client.0.smithi094.stdout: "rank": 0, 2021-07-26T06:13:49.713 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "a", 2021-07-26T06:13:49.713 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:49.713 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.714 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.714 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.714 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115", 2021-07-26T06:13:49.714 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 0 2021-07-26T06:13:49.715 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.715 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.715 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.715 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115/0", 2021-07-26T06:13:49.715 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:7115/0", 2021-07-26T06:13:49.716 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 0, 2021-07-26T06:13:49.716 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 0, 2021-07-26T06:13:49.716 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_location": "{}" 2021-07-26T06:13:49.716 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.716 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.717 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.717 INFO:tasks.workunit.client.0.smithi094.stdout: "quorum": [ 2021-07-26T06:13:49.717 INFO:tasks.workunit.client.0.smithi094.stdout: 0 2021-07-26T06:13:49.717 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.717 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap": { 2021-07-26T06:13:49.718 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 53, 2021-07-26T06:13:49.718 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:13:49.718 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:10:49.009557+0000", 2021-07-26T06:13:49.718 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:12:36.913606+0000", 2021-07-26T06:13:49.718 INFO:tasks.workunit.client.0.smithi094.stdout: "last_up_change": "2021-07-26T06:12:32.988643+0000", 2021-07-26T06:13:49.719 INFO:tasks.workunit.client.0.smithi094.stdout: "last_in_change": "2021-07-26T06:12:35.540674+0000", 2021-07-26T06:13:49.719 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-07-26T06:13:49.719 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_num": 5800384, 2021-07-26T06:13:49.719 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_set": [ 2021-07-26T06:13:49.719 INFO:tasks.workunit.client.0.smithi094.stdout: "nodown", 2021-07-26T06:13:49.720 INFO:tasks.workunit.client.0.smithi094.stdout: "noin", 2021-07-26T06:13:49.720 INFO:tasks.workunit.client.0.smithi094.stdout: "noout", 2021-07-26T06:13:49.720 INFO:tasks.workunit.client.0.smithi094.stdout: "pglog_hardlimit", 2021-07-26T06:13:49.720 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snapdirs", 2021-07-26T06:13:49.720 INFO:tasks.workunit.client.0.smithi094.stdout: "recovery_deletes", 2021-07-26T06:13:49.721 INFO:tasks.workunit.client.0.smithi094.stdout: "sortbitwise" 2021-07-26T06:13:49.721 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.721 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_version": 9, 2021-07-26T06:13:49.721 INFO:tasks.workunit.client.0.smithi094.stdout: "full_ratio": 0.99000000953674316, 2021-07-26T06:13:49.721 INFO:tasks.workunit.client.0.smithi094.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-07-26T06:13:49.722 INFO:tasks.workunit.client.0.smithi094.stdout: "nearfull_ratio": 0.85000002384185791, 2021-07-26T06:13:49.722 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_snapshot": "", 2021-07-26T06:13:49.722 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_max": 1, 2021-07-26T06:13:49.722 INFO:tasks.workunit.client.0.smithi094.stdout: "max_osd": 3, 2021-07-26T06:13:49.722 INFO:tasks.workunit.client.0.smithi094.stdout: "require_min_compat_client": "luminous", 2021-07-26T06:13:49.723 INFO:tasks.workunit.client.0.smithi094.stdout: "min_compat_client": "jewel", 2021-07-26T06:13:49.723 INFO:tasks.workunit.client.0.smithi094.stdout: "require_osd_release": "quincy", 2021-07-26T06:13:49.723 INFO:tasks.workunit.client.0.smithi094.stdout: "pools": [ 2021-07-26T06:13:49.723 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.724 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:13:49.724 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_name": "test", 2021-07-26T06:13:49.724 INFO:tasks.workunit.client.0.smithi094.stdout: "create_time": "2021-07-26T06:11:28.711595+0000", 2021-07-26T06:13:49.724 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": 1, 2021-07-26T06:13:49.724 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_names": "hashpspool", 2021-07-26T06:13:49.725 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:13:49.725 INFO:tasks.workunit.client.0.smithi094.stdout: "size": 3, 2021-07-26T06:13:49.725 INFO:tasks.workunit.client.0.smithi094.stdout: "min_size": 2, 2021-07-26T06:13:49.725 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_rule": 0, 2021-07-26T06:13:49.725 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_count": 0, 2021-07-26T06:13:49.726 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_target": 0, 2021-07-26T06:13:49.726 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_barrier": 0, 2021-07-26T06:13:49.726 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-07-26T06:13:49.726 INFO:tasks.workunit.client.0.smithi094.stdout: "object_hash": 2, 2021-07-26T06:13:49.726 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_autoscale_mode": "on", 2021-07-26T06:13:49.727 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num": 128, 2021-07-26T06:13:49.727 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num": 128, 2021-07-26T06:13:49.727 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num_target": 128, 2021-07-26T06:13:49.727 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_target": 128, 2021-07-26T06:13:49.727 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_pending": 128, 2021-07-26T06:13:49.728 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pg_merge_meta": { 2021-07-26T06:13:49.728 INFO:tasks.workunit.client.0.smithi094.stdout: "source_pgid": "0.0", 2021-07-26T06:13:49.728 INFO:tasks.workunit.client.0.smithi094.stdout: "ready_epoch": 0, 2021-07-26T06:13:49.728 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_started": 0, 2021-07-26T06:13:49.728 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": 0, 2021-07-26T06:13:49.729 INFO:tasks.workunit.client.0.smithi094.stdout: "source_version": "0'0", 2021-07-26T06:13:49.729 INFO:tasks.workunit.client.0.smithi094.stdout: "target_version": "0'0" 2021-07-26T06:13:49.729 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.729 INFO:tasks.workunit.client.0.smithi094.stdout: "last_change": "35", 2021-07-26T06:13:49.729 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend": "0", 2021-07-26T06:13:49.730 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_prenautilus": "0", 2021-07-26T06:13:49.730 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_preluminous": "33", 2021-07-26T06:13:49.730 INFO:tasks.workunit.client.0.smithi094.stdout: "auid": 0, 2021-07-26T06:13:49.730 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_mode": "selfmanaged", 2021-07-26T06:13:49.730 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_seq": 0, 2021-07-26T06:13:49.731 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_epoch": 0, 2021-07-26T06:13:49.731 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_snaps": [], 2021-07-26T06:13:49.731 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps": "[]", 2021-07-26T06:13:49.731 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_bytes": 0, 2021-07-26T06:13:49.731 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_objects": 0, 2021-07-26T06:13:49.732 INFO:tasks.workunit.client.0.smithi094.stdout: "tiers": [], 2021-07-26T06:13:49.732 INFO:tasks.workunit.client.0.smithi094.stdout: "tier_of": -1, 2021-07-26T06:13:49.732 INFO:tasks.workunit.client.0.smithi094.stdout: "read_tier": -1, 2021-07-26T06:13:49.732 INFO:tasks.workunit.client.0.smithi094.stdout: "write_tier": -1, 2021-07-26T06:13:49.733 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_mode": "none", 2021-07-26T06:13:49.733 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_bytes": 0, 2021-07-26T06:13:49.733 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_objects": 0, 2021-07-26T06:13:49.733 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-07-26T06:13:49.733 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-07-26T06:13:49.734 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_full_ratio_micro": 800000, 2021-07-26T06:13:49.734 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_flush_age": 0, 2021-07-26T06:13:49.734 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_evict_age": 0, 2021-07-26T06:13:49.734 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profile": "", 2021-07-26T06:13:49.735 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_params": { 2021-07-26T06:13:49.735 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "none" 2021-07-26T06:13:49.735 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.735 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_period": 0, 2021-07-26T06:13:49.735 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_count": 0, 2021-07-26T06:13:49.736 INFO:tasks.workunit.client.0.smithi094.stdout: "use_gmt_hitset": true, 2021-07-26T06:13:49.736 INFO:tasks.workunit.client.0.smithi094.stdout: "min_read_recency_for_promote": 0, 2021-07-26T06:13:49.736 INFO:tasks.workunit.client.0.smithi094.stdout: "min_write_recency_for_promote": 0, 2021-07-26T06:13:49.736 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_grade_decay_rate": 0, 2021-07-26T06:13:49.737 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_search_last_n": 0, 2021-07-26T06:13:49.737 INFO:tasks.workunit.client.0.smithi094.stdout: "grade_table": [], 2021-07-26T06:13:49.737 INFO:tasks.workunit.client.0.smithi094.stdout: "stripe_width": 0, 2021-07-26T06:13:49.737 INFO:tasks.workunit.client.0.smithi094.stdout: "expected_num_objects": 0, 2021-07-26T06:13:49.737 INFO:tasks.workunit.client.0.smithi094.stdout: "fast_read": false, 2021-07-26T06:13:49.738 INFO:tasks.workunit.client.0.smithi094.stdout: "options": {}, 2021-07-26T06:13:49.738 INFO:tasks.workunit.client.0.smithi094.stdout: "application_metadata": {} 2021-07-26T06:13:49.738 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.738 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.739 INFO:tasks.workunit.client.0.smithi094.stdout: "osds": [ 2021-07-26T06:13:49.739 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.739 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:49.739 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "b2ac31cc-94bd-49d4-89c7-dd5653384a9c", 2021-07-26T06:13:49.739 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:13:49.740 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:13:49.740 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:13:49.740 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:13:49.740 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 5, 2021-07-26T06:13:49.740 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 37, 2021-07-26T06:13:49.741 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 44, 2021-07-26T06:13:49.741 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 50, 2021-07-26T06:13:49.741 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 38, 2021-07-26T06:13:49.741 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:13:49.741 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:49.742 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.742 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.742 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.742 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6802", 2021-07-26T06:13:49.743 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.743 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.743 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.743 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.743 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6803", 2021-07-26T06:13:49.744 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.744 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.744 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.744 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.744 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:13:49.745 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.745 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.745 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.745 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6804", 2021-07-26T06:13:49.745 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.746 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.746 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.746 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.746 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6805", 2021-07-26T06:13:49.746 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.747 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.747 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.747 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.747 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:13:49.748 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.748 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.748 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.748 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6808", 2021-07-26T06:13:49.748 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.749 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.749 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.749 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.749 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6809", 2021-07-26T06:13:49.749 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.750 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.750 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.750 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.750 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:13:49.750 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.751 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.751 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.751 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6806", 2021-07-26T06:13:49.751 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.751 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.752 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.752 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.752 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6807", 2021-07-26T06:13:49.752 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52124 2021-07-26T06:13:49.753 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.753 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.753 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.753 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6803/52124", 2021-07-26T06:13:49.753 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6805/52124", 2021-07-26T06:13:49.754 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6809/52124", 2021-07-26T06:13:49.754 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6807/52124", 2021-07-26T06:13:49.754 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:13:49.754 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:13:49.755 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:13:49.755 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.755 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.755 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.755 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:49.756 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "64b2a2e5-8ab4-4197-9d59-4b39f363ac4c", 2021-07-26T06:13:49.756 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 0, 2021-07-26T06:13:49.756 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:13:49.756 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:13:49.756 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:13:49.757 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 0, 2021-07-26T06:13:49.757 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 0, 2021-07-26T06:13:49.757 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 10, 2021-07-26T06:13:49.757 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 35, 2021-07-26T06:13:49.757 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 39, 2021-07-26T06:13:49.758 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:13:49.758 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:49.758 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.758 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.759 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.759 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:13:49.759 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.759 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.759 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.760 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.760 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:13:49.760 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.760 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.760 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.761 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.761 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:13:49.761 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.761 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.761 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.762 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:13:49.762 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.762 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.762 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.763 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.763 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:13:49.763 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.763 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.763 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.764 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.764 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:13:49.764 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.764 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.764 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.765 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:13:49.765 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.765 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.765 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.765 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.766 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:13:49.766 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.766 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.766 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.766 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.767 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:13:49.767 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.767 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.767 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.767 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:13:49.768 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.768 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.768 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.768 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.768 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:13:49.769 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 46333 2021-07-26T06:13:49.769 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.769 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.769 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.769 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/46333", 2021-07-26T06:13:49.770 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/46333", 2021-07-26T06:13:49.770 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/46333", 2021-07-26T06:13:49.770 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/46333", 2021-07-26T06:13:49.770 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:13:49.770 INFO:tasks.workunit.client.0.smithi094.stdout: "exists" 2021-07-26T06:13:49.771 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.771 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.771 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.771 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:49.771 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "8f418200-be0a-4ee5-8381-ae936884bd8d", 2021-07-26T06:13:49.772 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:13:49.772 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:13:49.772 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:13:49.772 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:13:49.772 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 15, 2021-07-26T06:13:49.773 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 37, 2021-07-26T06:13:49.773 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 46, 2021-07-26T06:13:49.773 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 52, 2021-07-26T06:13:49.773 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 40, 2021-07-26T06:13:49.773 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:13:49.774 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:13:49.774 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.774 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.774 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.775 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:13:49.775 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.775 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.775 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.775 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.776 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:13:49.776 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.776 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.776 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.776 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.777 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:13:49.777 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.777 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.777 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.777 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:13:49.778 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.778 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.778 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.778 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.779 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:13:49.779 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.779 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.779 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.780 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.780 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:13:49.780 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.780 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.780 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.781 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:13:49.781 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.781 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.781 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.781 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.782 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:13:49.782 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.782 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.782 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.783 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.783 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:13:49.783 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:13:49.783 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.783 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:13:49.784 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:13:49.784 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.784 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.784 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.785 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:13:49.785 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:13:49.785 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 52704 2021-07-26T06:13:49.785 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.785 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.786 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.786 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/52704", 2021-07-26T06:13:49.786 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/52704", 2021-07-26T06:13:49.786 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/52704", 2021-07-26T06:13:49.787 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/52704", 2021-07-26T06:13:49.787 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:13:49.787 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:13:49.787 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:13:49.787 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.788 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.788 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.788 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_xinfo": [ 2021-07-26T06:13:49.788 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.789 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:49.789 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:12:21.330001+0000", 2021-07-26T06:13:49.789 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:13:49.789 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:13:49.789 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:13:49.790 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:13:49.790 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:11:00.710074+0000", 2021-07-26T06:13:49.790 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:13:49.790 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.790 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.791 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:49.791 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:12:21.858488+0000", 2021-07-26T06:13:49.791 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:13:49.791 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:13:49.792 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:13:49.792 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:13:49.792 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:11:12.450854+0000", 2021-07-26T06:13:49.792 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:13:49.792 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.793 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.793 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:49.793 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:12:22.519390+0000", 2021-07-26T06:13:49.793 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:13:49.794 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:13:49.794 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:13:49.794 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:13:49.794 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:11:25.473164+0000", 2021-07-26T06:13:49.795 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:13:49.795 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.795 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.795 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap": [], 2021-07-26T06:13:49.795 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap_items": [], 2021-07-26T06:13:49.796 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_temp": [], 2021-07-26T06:13:49.796 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_temp": [], 2021-07-26T06:13:49.796 INFO:tasks.workunit.client.0.smithi094.stdout: "blocklist": {}, 2021-07-26T06:13:49.796 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profiles": { 2021-07-26T06:13:49.797 INFO:tasks.workunit.client.0.smithi094.stdout: "default": { 2021-07-26T06:13:49.797 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:13:49.797 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "2", 2021-07-26T06:13:49.797 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:13:49.797 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van" 2021-07-26T06:13:49.798 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.798 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.798 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps_queue": [], 2021-07-26T06:13:49.798 INFO:tasks.workunit.client.0.smithi094.stdout: "new_removed_snaps": [], 2021-07-26T06:13:49.798 INFO:tasks.workunit.client.0.smithi094.stdout: "new_purged_snaps": [], 2021-07-26T06:13:49.799 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_node_flags": {}, 2021-07-26T06:13:49.799 INFO:tasks.workunit.client.0.smithi094.stdout: "device_class_flags": {}, 2021-07-26T06:13:49.799 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": { 2021-07-26T06:13:49.799 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_enabled": false, 2021-07-26T06:13:49.800 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_bucket_count": 0, 2021-07-26T06:13:49.800 INFO:tasks.workunit.client.0.smithi094.stdout: "degraded_stretch_mode": 0, 2021-07-26T06:13:49.800 INFO:tasks.workunit.client.0.smithi094.stdout: "recovering_stretch_mode": 0, 2021-07-26T06:13:49.800 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_bucket": 0 2021-07-26T06:13:49.800 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.801 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.801 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_metadata": [ 2021-07-26T06:13:49.801 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.801 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:49.802 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:13:49.802 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6804/52124,v1:127.0.0.1:6805/52124]", 2021-07-26T06:13:49.802 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:13:49.802 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:13:49.802 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:13:49.803 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:13:49.803 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:13:49.803 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:13:49.803 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:13:49.804 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:13:49.804 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:13:49.804 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-07-26T06:13:49.804 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:13:49.804 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:13:49.805 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:13:49.805 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:13:49.805 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:13:49.805 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:13:49.805 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:13:49.806 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:13:49.806 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:13:49.806 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:13:49.806 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:13:49.806 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:13:49.807 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:13:49.807 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:13:49.807 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:13:49.807 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6802/52124,v1:127.0.0.1:6803/52124]", 2021-07-26T06:13:49.808 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:13:49.808 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/52124,v1:127.0.0.1:6809/52124]", 2021-07-26T06:13:49.808 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/52124,v1:127.0.0.1:6807/52124]", 2021-07-26T06:13:49.808 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:13:49.808 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:13:49.809 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:13:49.809 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:13:49.809 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:13:49.809 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:13:49.809 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:13:49.810 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:13:49.810 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:13:49.810 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/0", 2021-07-26T06:13:49.810 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:13:49.811 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:13:49.811 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:13:49.811 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.811 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.811 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:49.812 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:13:49.812 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/46333,v1:127.0.0.1:6813/46333]", 2021-07-26T06:13:49.812 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:13:49.812 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:13:49.812 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:13:49.813 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:13:49.813 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:13:49.813 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:13:49.813 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:13:49.814 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:13:49.814 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:13:49.814 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-07-26T06:13:49.814 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:13:49.814 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:13:49.815 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:13:49.815 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:13:49.815 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:13:49.815 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:13:49.816 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:13:49.816 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:13:49.816 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:13:49.816 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:13:49.816 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:13:49.817 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:13:49.817 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:13:49.817 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:13:49.817 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:13:49.817 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/46333,v1:127.0.0.1:6811/46333]", 2021-07-26T06:13:49.818 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:13:49.818 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/46333,v1:127.0.0.1:6817/46333]", 2021-07-26T06:13:49.818 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/46333,v1:127.0.0.1:6815/46333]", 2021-07-26T06:13:49.818 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:13:49.819 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:13:49.819 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:13:49.819 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:13:49.819 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:13:49.819 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:13:49.820 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:13:49.820 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:13:49.820 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:13:49.820 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/1", 2021-07-26T06:13:49.820 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:13:49.821 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:13:49.821 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:13:49.821 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.821 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.822 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:49.822 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:13:49.822 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/52704,v1:127.0.0.1:6813/52704]", 2021-07-26T06:13:49.822 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:13:49.822 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:13:49.823 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:13:49.823 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:13:49.823 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:13:49.823 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:13:49.824 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:13:49.824 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:13:49.824 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:13:49.824 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-07-26T06:13:49.824 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:13:49.825 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:13:49.825 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:13:49.825 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:13:49.825 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:13:49.825 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:13:49.826 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:13:49.826 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:13:49.826 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:13:49.826 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:13:49.827 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:13:49.827 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:13:49.827 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:13:49.827 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:13:49.827 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:13:49.828 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/52704,v1:127.0.0.1:6811/52704]", 2021-07-26T06:13:49.828 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:13:49.828 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/52704,v1:127.0.0.1:6817/52704]", 2021-07-26T06:13:49.828 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/52704,v1:127.0.0.1:6815/52704]", 2021-07-26T06:13:49.828 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:13:49.829 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:13:49.829 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:13:49.829 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:13:49.829 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:13:49.830 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:13:49.830 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:13:49.830 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:13:49.830 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:13:49.830 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/2", 2021-07-26T06:13:49.831 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:13:49.831 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:13:49.831 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:13:49.831 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.832 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.832 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_clean_epochs": { 2021-07-26T06:13:49.832 INFO:tasks.workunit.client.0.smithi094.stdout: "min_last_epoch_clean": 0, 2021-07-26T06:13:49.832 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": { 2021-07-26T06:13:49.832 INFO:tasks.workunit.client.0.smithi094.stdout: "per_pool": [ 2021-07-26T06:13:49.833 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.833 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:13:49.833 INFO:tasks.workunit.client.0.smithi094.stdout: "floor": 36 2021-07-26T06:13:49.833 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.833 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.834 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.834 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_epochs": [ 2021-07-26T06:13:49.834 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.834 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:49.835 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 44 2021-07-26T06:13:49.840 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.841 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.841 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:49.841 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 10 2021-07-26T06:13:49.841 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.842 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.842 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:49.842 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 46 2021-07-26T06:13:49.842 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.843 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.843 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.843 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_first_committed": 1, 2021-07-26T06:13:49.843 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_last_committed": 53, 2021-07-26T06:13:49.843 INFO:tasks.workunit.client.0.smithi094.stdout: "crushmap": { 2021-07-26T06:13:49.844 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": [ 2021-07-26T06:13:49.844 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.844 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:49.844 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.0", 2021-07-26T06:13:49.845 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:13:49.845 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.845 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.845 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:49.845 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.1", 2021-07-26T06:13:49.846 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:13:49.846 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.846 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.846 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:49.846 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.2", 2021-07-26T06:13:49.847 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:13:49.847 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.847 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.847 INFO:tasks.workunit.client.0.smithi094.stdout: "types": [ 2021-07-26T06:13:49.848 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.848 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 0, 2021-07-26T06:13:49.848 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd" 2021-07-26T06:13:49.848 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.848 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.849 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:13:49.849 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "host" 2021-07-26T06:13:49.849 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.849 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.849 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 2, 2021-07-26T06:13:49.850 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "chassis" 2021-07-26T06:13:49.850 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.850 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.850 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 3, 2021-07-26T06:13:49.850 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "rack" 2021-07-26T06:13:49.851 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.851 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.851 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 4, 2021-07-26T06:13:49.851 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "row" 2021-07-26T06:13:49.852 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.852 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.852 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 5, 2021-07-26T06:13:49.852 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pdu" 2021-07-26T06:13:49.852 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.853 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.853 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 6, 2021-07-26T06:13:49.853 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pod" 2021-07-26T06:13:49.853 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.853 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.854 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 7, 2021-07-26T06:13:49.854 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "room" 2021-07-26T06:13:49.854 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.854 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.854 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 8, 2021-07-26T06:13:49.855 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "datacenter" 2021-07-26T06:13:49.855 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.855 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.855 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 9, 2021-07-26T06:13:49.856 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "zone" 2021-07-26T06:13:49.856 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.856 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.856 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 10, 2021-07-26T06:13:49.856 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "region" 2021-07-26T06:13:49.857 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.857 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.857 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:13:49.857 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "root" 2021-07-26T06:13:49.857 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.858 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.858 INFO:tasks.workunit.client.0.smithi094.stdout: "buckets": [ 2021-07-26T06:13:49.858 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.858 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -1, 2021-07-26T06:13:49.858 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default", 2021-07-26T06:13:49.859 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:13:49.859 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:13:49.859 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:49.859 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:49.860 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:49.860 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:49.860 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.860 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:13:49.860 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:49.861 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:49.861 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.861 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.861 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.862 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.862 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -2, 2021-07-26T06:13:49.862 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default~hdd", 2021-07-26T06:13:49.862 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:13:49.862 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:13:49.863 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:49.863 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:49.863 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:49.863 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:49.863 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.864 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:13:49.864 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:49.864 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:49.864 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.865 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.865 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.865 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.865 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:13:49.865 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094", 2021-07-26T06:13:49.866 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:13:49.866 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:13:49.866 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:49.866 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:49.867 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:49.867 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:49.867 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.867 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:49.867 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:49.868 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:49.868 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.868 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.868 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:49.868 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:49.869 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:13:49.869 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.869 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.869 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:49.869 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:49.870 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:13:49.870 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.870 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.870 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.871 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.871 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:13:49.871 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094~hdd", 2021-07-26T06:13:49.871 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:13:49.871 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:13:49.872 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:13:49.872 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:13:49.872 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:13:49.872 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:13:49.872 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.873 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:13:49.873 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:49.873 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:13:49.873 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.873 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.874 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:13:49.874 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:49.874 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:13:49.874 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.875 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.875 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:13:49.875 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:13:49.875 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:13:49.875 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.876 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.876 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.876 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.876 INFO:tasks.workunit.client.0.smithi094.stdout: "rules": [ 2021-07-26T06:13:49.876 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.877 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 0, 2021-07-26T06:13:49.877 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "replicated_rule", 2021-07-26T06:13:49.877 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:13:49.877 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:13:49.877 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.878 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:13:49.878 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:13:49.878 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:13:49.878 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.878 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.879 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_firstn", 2021-07-26T06:13:49.879 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:13:49.879 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:13:49.879 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.879 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.880 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:13:49.880 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.880 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:13:49.880 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.880 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.881 INFO:tasks.workunit.client.0.smithi094.stdout: "tunables": { 2021-07-26T06:13:49.881 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_tries": 0, 2021-07-26T06:13:49.881 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_fallback_tries": 0, 2021-07-26T06:13:49.881 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_total_tries": 50, 2021-07-26T06:13:49.881 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_descend_once": 1, 2021-07-26T06:13:49.882 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_vary_r": 1, 2021-07-26T06:13:49.882 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_stable": 1, 2021-07-26T06:13:49.882 INFO:tasks.workunit.client.0.smithi094.stdout: "straw_calc_version": 1, 2021-07-26T06:13:49.882 INFO:tasks.workunit.client.0.smithi094.stdout: "allowed_bucket_algs": 54, 2021-07-26T06:13:49.882 INFO:tasks.workunit.client.0.smithi094.stdout: "profile": "jewel", 2021-07-26T06:13:49.883 INFO:tasks.workunit.client.0.smithi094.stdout: "optimal_tunables": 1, 2021-07-26T06:13:49.883 INFO:tasks.workunit.client.0.smithi094.stdout: "legacy_tunables": 0, 2021-07-26T06:13:49.883 INFO:tasks.workunit.client.0.smithi094.stdout: "minimum_required_version": "jewel", 2021-07-26T06:13:49.883 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables": 1, 2021-07-26T06:13:49.883 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables2": 1, 2021-07-26T06:13:49.884 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v2_rules": 0, 2021-07-26T06:13:49.884 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables3": 1, 2021-07-26T06:13:49.884 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v3_rules": 0, 2021-07-26T06:13:49.884 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v4_buckets": 1, 2021-07-26T06:13:49.885 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables5": 1, 2021-07-26T06:13:49.885 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v5_rules": 0 2021-07-26T06:13:49.885 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.885 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_args": {} 2021-07-26T06:13:49.885 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.886 INFO:tasks.workunit.client.0.smithi094.stdout: "fsmap": { 2021-07-26T06:13:49.886 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:13:49.886 INFO:tasks.workunit.client.0.smithi094.stdout: "default_fscid": -1, 2021-07-26T06:13:49.886 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": { 2021-07-26T06:13:49.886 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": {}, 2021-07-26T06:13:49.887 INFO:tasks.workunit.client.0.smithi094.stdout: "ro_compat": {}, 2021-07-26T06:13:49.887 INFO:tasks.workunit.client.0.smithi094.stdout: "incompat": { 2021-07-26T06:13:49.887 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_1": "base v0.20", 2021-07-26T06:13:49.887 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_2": "client writeable ranges", 2021-07-26T06:13:49.887 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_3": "default file layouts on dirs", 2021-07-26T06:13:49.888 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_4": "dir inode in separate object", 2021-07-26T06:13:49.888 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_5": "mds uses versioned encoding", 2021-07-26T06:13:49.888 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_6": "dirfrag is stored in omap", 2021-07-26T06:13:49.888 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_8": "no anchor table", 2021-07-26T06:13:49.888 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_9": "file layout v2", 2021-07-26T06:13:49.889 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_10": "snaprealm v2" 2021-07-26T06:13:49.889 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.889 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.889 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_flags": { 2021-07-26T06:13:49.889 INFO:tasks.workunit.client.0.smithi094.stdout: "enable_multiple": true, 2021-07-26T06:13:49.890 INFO:tasks.workunit.client.0.smithi094.stdout: "ever_enabled_multiple": true 2021-07-26T06:13:49.890 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.890 INFO:tasks.workunit.client.0.smithi094.stdout: "standbys": [], 2021-07-26T06:13:49.890 INFO:tasks.workunit.client.0.smithi094.stdout: "filesystems": [] 2021-07-26T06:13:49.890 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.891 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_first_committed": 1, 2021-07-26T06:13:49.891 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_last_committed": 1, 2021-07-26T06:13:49.891 INFO:tasks.workunit.client.0.smithi094.stdout: "auth": { 2021-07-26T06:13:49.891 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:13:49.891 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 5, 2021-07-26T06:13:49.892 INFO:tasks.workunit.client.0.smithi094.stdout: "num_secrets": 3 2021-07-26T06:13:49.892 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.892 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 128, 2021-07-26T06:13:49.892 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_active": 128, 2021-07-26T06:13:49.892 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_unknown": 0, 2021-07-26T06:13:49.893 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osd": 3, 2021-07-26T06:13:49.893 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_sum": { 2021-07-26T06:13:49.893 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:13:49.893 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 62021, 2021-07-26T06:13:49.893 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 99, 2021-07-26T06:13:49.894 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:13:49.894 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 297, 2021-07-26T06:13:49.894 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:13:49.894 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:13:49.895 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 99, 2021-07-26T06:13:49.895 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:13:49.895 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:13:49.895 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 99, 2021-07-26T06:13:49.895 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:13:49.896 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:13:49.896 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:13:49.896 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 95, 2021-07-26T06:13:49.896 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 95, 2021-07-26T06:13:49.897 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:13:49.897 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:13:49.897 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:13:49.897 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:13:49.897 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:13:49.897 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:13:49.898 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:13:49.898 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:13:49.898 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:13:49.898 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:13:49.899 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:13:49.899 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:13:49.899 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:13:49.899 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:13:49.899 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:13:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:13:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:13:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:13:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:13:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:13:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:13:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:13:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:13:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:13:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:13:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:13:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:13:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:13:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 0, 2021-07-26T06:13:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 0, 2021-07-26T06:13:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:13:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:13:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 2182, 2021-07-26T06:13:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 2182, 2021-07-26T06:13:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 256, 2021-07-26T06:13:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 256, 2021-07-26T06:13:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 0 2021-07-26T06:13:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum": { 2021-07-26T06:13:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:13:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:13:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 384, 2021-07-26T06:13:49.907 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:13:49.907 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:13:49.907 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:13:49.907 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:13:49.907 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 24616, 2021-07-26T06:13:49.908 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2384, 2021-07-26T06:13:49.908 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:13:49.908 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 22205, 2021-07-26T06:13:49.908 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314548184, 2021-07-26T06:13:49.908 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:13:49.909 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:13:49.909 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322097340416, 2021-07-26T06:13:49.909 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:49.909 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2441216, 2021-07-26T06:13:49.909 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 590043, 2021-07-26T06:13:49.910 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:49.910 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:49.910 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:49.910 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:13:49.911 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 22738832 2021-07-26T06:13:49.911 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.911 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:13:49.911 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:13:49.911 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:13:49.912 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:13:49.912 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:13:49.912 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:13:49.912 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:13:49.912 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.913 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:13:49.913 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 50, 2021-07-26T06:13:49.913 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 50, 2021-07-26T06:13:49.913 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 50000000, 2021-07-26T06:13:49.913 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 50000000 2021-07-26T06:13:49.914 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.914 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:13:49.914 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:13:49.914 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.915 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum_by_class": { 2021-07-26T06:13:49.915 INFO:tasks.workunit.client.0.smithi094.stdout: "hdd": { 2021-07-26T06:13:49.915 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:13:49.915 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:13:49.915 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 384, 2021-07-26T06:13:49.916 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:13:49.916 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:13:49.916 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:13:49.916 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:13:49.916 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 24616, 2021-07-26T06:13:49.917 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2384, 2021-07-26T06:13:49.917 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:13:49.917 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 22205, 2021-07-26T06:13:49.917 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314548184, 2021-07-26T06:13:49.918 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:13:49.918 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:13:49.918 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322097340416, 2021-07-26T06:13:49.918 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:49.918 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2441216, 2021-07-26T06:13:49.919 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 590043, 2021-07-26T06:13:49.919 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:49.919 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:49.919 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:49.920 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:13:49.920 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 22738832 2021-07-26T06:13:49.920 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.920 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:13:49.920 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:13:49.921 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:13:49.921 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:13:49.921 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:13:49.921 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:13:49.921 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:13:49.922 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.922 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:13:49.922 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 50, 2021-07-26T06:13:49.922 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 50, 2021-07-26T06:13:49.922 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 50000000, 2021-07-26T06:13:49.923 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 50000000 2021-07-26T06:13:49.923 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.923 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:13:49.923 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:13:49.923 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.924 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.924 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_stats": [ 2021-07-26T06:13:49.924 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.924 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:13:49.924 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 128, 2021-07-26T06:13:49.925 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:13:49.925 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 62021, 2021-07-26T06:13:49.925 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 99, 2021-07-26T06:13:49.925 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 0, 2021-07-26T06:13:49.925 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 297, 2021-07-26T06:13:49.926 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:13:49.926 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:13:49.926 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 99, 2021-07-26T06:13:49.926 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:13:49.927 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:13:49.927 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 99, 2021-07-26T06:13:49.927 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:13:49.927 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:13:49.927 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:13:49.928 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 95, 2021-07-26T06:13:49.928 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 95, 2021-07-26T06:13:49.928 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:13:49.928 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:13:49.928 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:13:49.929 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:13:49.929 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:13:49.929 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:13:49.929 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:13:49.930 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:13:49.930 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:13:49.930 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:13:49.930 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:13:49.930 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:13:49.931 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:13:49.931 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:13:49.931 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:13:49.931 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:13:49.931 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:13:49.932 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:13:49.932 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:13:49.932 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:13:49.932 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:13:49.933 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:13:49.933 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:13:49.933 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:13:49.933 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:13:49.933 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.934 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:13:49.934 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:13:49.934 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:13:49.934 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:13:49.935 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 1220608, 2021-07-26T06:13:49.935 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 184438, 2021-07-26T06:13:49.935 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:13:49.935 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:13:49.935 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:13:49.936 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:13:49.936 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:13:49.936 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.936 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 2182, 2021-07-26T06:13:49.937 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 2182, 2021-07-26T06:13:49.937 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 256, 2021-07-26T06:13:49.937 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 256, 2021-07-26T06:13:49.937 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 3 2021-07-26T06:13:49.937 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.938 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.938 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_stats": [ 2021-07-26T06:13:49.938 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.938 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:49.938 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 188978561044 2021-07-26T06:13:49.939 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.939 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.939 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:49.939 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 42949672981 2021-07-26T06:13:49.939 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.940 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.940 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:49.940 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 197568495635 2021-07-26T06:13:49.940 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.940 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.941 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_state": [ 2021-07-26T06:13:49.941 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.941 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized+degraded", 2021-07-26T06:13:49.941 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 93 2021-07-26T06:13:49.942 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.942 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.942 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized", 2021-07-26T06:13:49.942 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 35 2021-07-26T06:13:49.942 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.943 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.943 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_osd": [ 2021-07-26T06:13:49.943 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.943 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:13:49.944 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 59, 2021-07-26T06:13:49.944 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 128, 2021-07-26T06:13:49.944 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:13:49.944 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.944 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.945 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:13:49.945 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 69, 2021-07-26T06:13:49.945 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 128, 2021-07-26T06:13:49.945 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:13:49.945 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.946 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.946 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:13:49.946 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:13:49.946 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 0, 2021-07-26T06:13:49.947 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:13:49.947 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.947 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.947 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": [ 2021-07-26T06:13:49.947 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:13:49.948 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:13:49.948 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": {} 2021-07-26T06:13:49.948 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.948 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:13:49.949 INFO:tasks.workunit.client.0.smithi094.stdout: "servicemap": { 2021-07-26T06:13:49.949 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 7, 2021-07-26T06:13:49.949 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:13:33.865803+0000", 2021-07-26T06:13:49.949 INFO:tasks.workunit.client.0.smithi094.stdout: "services": {} 2021-07-26T06:13:49.949 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:13:49.950 INFO:tasks.workunit.client.0.smithi094.stdout: "paxos": { 2021-07-26T06:13:49.950 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:13:49.950 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 537, 2021-07-26T06:13:49.950 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pn": 0, 2021-07-26T06:13:49.951 INFO:tasks.workunit.client.0.smithi094.stdout: "accepted_pn": 0 2021-07-26T06:13:49.951 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:13:49.951 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:13:49.952 INFO:tasks.workunit.client.0.smithi094.stderr:report 72428926 2021-07-26T06:13:49.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: return 1 2021-07-26T06:13:49.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2021-07-26T06:13:49.953 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery 2021-07-26T06:13:49.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:13:50.053 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:13:50.053 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.656870+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.054 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061855+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.054 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.056238+0000 34'96 52:135 [0,2] 0 [0,2] 0 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.054 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059226+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.054 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 1 0 0 617 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651961+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.055 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055365+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.055 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 1 0 0 617 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650564+0000 34'96 53:145 [2,0] 2 [2,0] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.055 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 3 0 0 1851 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651681+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.055 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161682+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.056 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056354+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.056 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063687+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.056 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060493+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.056 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647254+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.056 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649803+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.057 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650413+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.057 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064477+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.057 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.648792+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:13:50.057 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649585+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.057 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061742+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.058 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648454+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.058 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649644+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.058 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645510+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.058 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:36.981078+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.059 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054380+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.059 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645644+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.059 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059874+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.059 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.656499+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.060 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061665+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.060 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.165344+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.060 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057712+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.060 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160614+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.060 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.655228+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.061 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652396+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.061 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652639+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.061 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647082+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.061 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161648+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.062 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644788+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.062 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161366+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.062 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650536+0000 34'96 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.062 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057570+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.063 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062423+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.063 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061857+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.063 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651257+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.063 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.650322+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.063 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064099+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.064 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060925+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.064 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646360+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.064 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 1 0 0 601 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.655464+0000 53'96 53:118 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.064 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648831+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.065 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059519+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.065 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055783+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.065 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058878+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.065 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055308+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.065 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646801+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.066 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057260+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.066 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059379+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.066 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.056265+0000 32'95 52:138 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.066 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649448+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.067 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055108+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.067 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646439+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.067 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056459+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.067 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.059557+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.067 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058386+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.068 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060541+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.068 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651884+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.068 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063303+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.068 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060893+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.069 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652020+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.069 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063243+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.069 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651093+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.069 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644530+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.069 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061408+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.070 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.070 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.070 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061302+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.070 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159501+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.070 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160463+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.071 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.654813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.071 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063624+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.071 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652317+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.071 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062782+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.072 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651456+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.072 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159722+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.072 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648810+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.072 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652809+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.072 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651444+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.073 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054311+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.073 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160718+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.073 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650851+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.073 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058230+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.074 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651009+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.074 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.163179+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.074 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.592753+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.074 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064015+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.075 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647935+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.075 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064861+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.075 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.159494+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.075 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652699+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.075 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.159523+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.076 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.062968+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.076 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647259+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.076 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.162249+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.076 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.056263+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.077 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645544+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.077 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.054889+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.077 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.161101+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.077 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644646+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.077 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.656444+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.078 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.058220+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.078 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.059923+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.078 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645471+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.078 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652814+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.078 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.079 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652308+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.079 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644586+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.079 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648452+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.079 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.646610+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.079 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.057203+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.080 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648029+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.080 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655913+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.080 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.060941+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.080 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655011+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.081 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.645795+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.081 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.054246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.081 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058839+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.081 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.162128+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.081 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058635+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.082 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060058+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.082 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:13:50.082 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:13:50.083 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:13:50.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:13:50.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2021-07-26T06:13:50.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:13:50.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:13:50.085 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:13:50.085 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:13:50.085 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:13:50.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:13:50.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-07-26T06:13:50.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok set_recovery_delay 100000 2021-07-26T06:13:50.153 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:13:50.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-07-26T06:13:50.161 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2021-07-26T06:13:50.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:13:50.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:13:50.163 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:13:50.163 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:13:50.163 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:13:50.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:13:50.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-07-26T06:13:50.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok set_recovery_delay 100000 2021-07-26T06:13:50.245 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:13:50.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2021-07-26T06:13:50.253 INFO:tasks.workunit.client.0.smithi094.stdout:revive divergent 1 2021-07-26T06:13:50.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:13:50.609 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:13:50.609 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.656870+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.610 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061855+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.610 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.056238+0000 34'96 52:135 [0,2] 0 [0,2] 0 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.611 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059226+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.611 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 1 0 0 617 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651961+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.612 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055365+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.612 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 1 0 0 617 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650564+0000 34'96 53:145 [2,0] 2 [2,0] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.612 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 3 0 0 1851 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651681+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.612 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161682+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.613 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056354+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.613 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063687+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.613 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060493+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.614 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647254+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.614 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649803+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.614 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650413+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.614 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064477+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.615 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.648792+0000 32'95 53:133 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:13:50.615 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649585+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.615 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061742+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.615 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648454+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.615 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649644+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.616 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645510+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.616 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:36.981078+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.616 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054380+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.616 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645644+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.617 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059874+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.617 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.656499+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.617 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061665+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.617 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.165344+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.617 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057712+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.618 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160614+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.618 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.655228+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.618 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652396+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.618 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652639+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.619 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647082+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.619 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161648+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.619 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644788+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.620 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161366+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.620 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650536+0000 34'96 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.620 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057570+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.621 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062423+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.621 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061857+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.621 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651257+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.621 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.650322+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.622 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064099+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.622 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060925+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.622 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646360+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.622 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 1 0 0 601 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.655464+0000 53'96 53:118 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.623 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648831+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.623 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059519+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.623 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055783+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.623 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058878+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.623 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055308+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.624 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646801+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.624 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057260+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.624 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059379+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.624 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.056265+0000 32'95 52:138 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.625 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649448+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.625 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055108+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.625 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646439+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.625 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056459+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.625 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.059557+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.626 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058386+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.626 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060541+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.626 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651884+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.626 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063303+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.627 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060893+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.627 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652020+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.627 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063243+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.627 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651093+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.628 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644530+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.628 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061408+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.628 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.628 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.628 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061302+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.629 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159501+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.629 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160463+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.629 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.654813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.629 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063624+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.630 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652317+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.630 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062782+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.630 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651456+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.631 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159722+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.631 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648810+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.631 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652809+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.631 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651444+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.632 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054311+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.632 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160718+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.632 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650851+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.632 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058230+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.632 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651009+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.633 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.163179+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.633 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.592753+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.633 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064015+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.633 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647935+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.634 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064861+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.634 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.159494+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.634 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652699+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.635 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.159523+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.635 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.062968+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.635 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647259+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.635 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.162249+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.635 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.056263+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.636 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645544+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.636 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.054889+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.636 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.161101+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.636 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644646+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.637 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.656444+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.637 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.058220+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.637 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.059923+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.637 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645471+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.638 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652814+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.638 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647813+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.638 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652308+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.639 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644586+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.639 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648452+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.639 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.646610+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.640 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.057203+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.640 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648029+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.640 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655913+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.641 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.060941+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.641 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655011+0000 32'95 53:129 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.641 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.645795+0000 32'95 53:117 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.641 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.054246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.641 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058839+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.642 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.162128+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.642 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058635+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.642 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060058+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:50.642 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:13:50.643 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:13:50.643 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:13:50.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2021-07-26T06:13:51.442 INFO:tasks.workunit.client.0.smithi094.stderr:noup is set 2021-07-26T06:13:51.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-07-26T06:13:51.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:13:51.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:13:51.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:13:51.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:13:51.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:13:51.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:13:51.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:13:51.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:13:51.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:13:51.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:13:51.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:13:51.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:13:51.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:13:51.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:13:51.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:13:51.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:13:51.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:13:51.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:13:51.461 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:13:51.461 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:13:51.462 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:13:51.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:13:51.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:13:51.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:13:51.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:13:51.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:13:51.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:13:51.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:13:51.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:13:51.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:13:51.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:13:51.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:13:51.466 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:13:51.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:13:51.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:13:51.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:13:51.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:13:51.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:13:51.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:13:51.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:13:51.498 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:13:51.493+0000 7f6508d9df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:51.506 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:13:51.501+0000 7f6508d9df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:51.508 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:13:51.505+0000 7f6508d9df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:13:51.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: activate_osd: return 0 2021-07-26T06:13:51.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2021-07-26T06:13:52.050 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:13:52.045+0000 7f6508d9df40 -1 Falling back to public interface 2021-07-26T06:13:55.452 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:13:55.445+0000 7f6508d9df40 -1 osd.1 37 log_to_monitors {default=true} 2021-07-26T06:13:56.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2021-07-26T06:13:56.848 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery divergent 2021-07-26T06:13:56.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:13:57.214 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:13:57.214 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.656870+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.214 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061855+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.056238+0000 34'96 52:135 [0,2] 0 [0,2] 0 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059226+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.215 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 1 0 0 617 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651961+0000 32'95 54:135 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055365+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 1 0 0 617 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650564+0000 34'96 54:147 [2,0] 2 [2,0] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 3 0 0 1851 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651681+0000 32'95 54:135 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.216 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161682+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056354+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063687+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060493+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.217 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647254+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649803+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650413+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064477+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.648792+0000 32'95 54:135 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:13:57.218 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649585+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061742+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648454+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649644+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.219 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645510+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:36.981078+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054380+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645644+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059874+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.220 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.656499+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061665+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.165344+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057712+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.221 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160614+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.655228+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652396+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652639+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.222 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647082+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161648+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644788+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161366+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.223 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650536+0000 34'96 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057570+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062423+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061857+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651257+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.224 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.650322+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064099+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060925+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646360+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 1 0 0 601 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.655464+0000 53'96 54:120 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648831+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059519+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055783+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058878+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.226 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055308+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646801+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057260+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059379+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.227 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.056265+0000 32'95 52:138 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649448+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055108+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646439+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056459+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.228 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.059557+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058386+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060541+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651884+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063303+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060893+0000 32'95 52:123 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652020+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063243+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.230 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651093+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.231 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644530+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.231 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061408+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.231 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650813+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061302+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159501+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.232 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160463+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.654813+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063624+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652317+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062782+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.233 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651456+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159722+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648810+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652809+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651444+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.234 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054311+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160718+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650851+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058230+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.235 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651009+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.163179+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.592753+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064015+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.236 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647935+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064861+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.159494+0000 34'96 52:146 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652699+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.238 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.159523+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.238 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.062968+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.239 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647259+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.239 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.162249+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.239 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.056263+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.240 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645544+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.240 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.054889+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.241 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.161101+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.241 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644646+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.241 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.656444+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.242 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.058220+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.242 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.059923+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.243 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645471+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.243 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652814+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.243 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647813+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.244 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652308+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.244 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644586+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.245 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648452+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.245 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.646610+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.245 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.057203+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.246 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648029+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.246 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655913+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.246 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.060941+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.247 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655011+0000 32'95 54:131 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.247 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.645795+0000 32'95 54:119 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.247 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.054246+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.247 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058839+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.248 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.162128+0000 32'95 52:123 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.248 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058635+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.248 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060058+0000 32'95 52:134 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:13:57.248 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:13:57.249 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:13:57.250 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:13:57.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2021-07-26T06:13:57.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:13:57.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:13:57.251 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:13:57.251 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:13:57.252 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:13:57.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:13:57.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2021-07-26T06:13:57.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.1.asok set_recovery_delay 100000 2021-07-26T06:13:57.320 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:13:57.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2021-07-26T06:13:58.195 INFO:tasks.workunit.client.0.smithi094.stderr:noup is unset 2021-07-26T06:13:58.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2021-07-26T06:13:58.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:13:58.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:13:58.208 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:13:58.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:13:58.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:13:58.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:58.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:13:58.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:58.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:13:58.959 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 44 up_thru 50 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/52124,v1:127.0.0.1:6803/52124] [v2:127.0.0.1:6804/52124,v1:127.0.0.1:6805/52124] exists,up b2ac31cc-94bd-49d4-89c7-dd5653384a9c 2021-07-26T06:13:58.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:13:58.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:13:58.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:13:58.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2021-07-26T06:13:58.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:13:58.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:13:58.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:13:58.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:13:58.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:58.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:13:58.963 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:13:58.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:58.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:13:59.485 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 39 last_clean_interval [10,37) [v2:127.0.0.1:6818/55407,v1:127.0.0.1:6819/55407] [v2:127.0.0.1:6820/55407,v1:127.0.0.1:6821/55407] exists,up 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c 2021-07-26T06:13:59.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:13:59.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:13:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:13:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2021-07-26T06:13:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:13:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:13:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:13:59.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:13:59.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:13:59.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:13:59.490 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:13:59.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:13:59.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:14:00.077 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 46 up_thru 58 down_at 40 last_clean_interval [15,37) [v2:127.0.0.1:6810/52704,v1:127.0.0.1:6811/52704] [v2:127.0.0.1:6812/52704,v1:127.0.0.1:6813/52704] exists,up 8f418200-be0a-4ee5-8381-ae936884bd8d 2021-07-26T06:14:00.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:14:00.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:14:00.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:14:00.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:14:00.444 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:14:00.444 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.656870+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.445 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061855+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.445 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.056238+0000 34'96 55:137 [0,2] 0 [0,2] 0 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.445 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059226+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.445 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 1 0 0 617 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651961+0000 32'95 56:137 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.445 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055365+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.446 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 1 0 0 617 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650564+0000 34'96 56:149 [2,0] 2 [2,0] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.446 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 3 0 0 1851 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651681+0000 32'95 56:137 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.446 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161682+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.446 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056354+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.447 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063687+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.447 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060493+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.447 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647254+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.447 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649803+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.447 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650413+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.448 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064477+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.448 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.648792+0000 32'95 56:137 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:14:00.448 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649585+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.448 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061742+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.449 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648454+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.449 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649644+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.449 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645510+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.449 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:36.981078+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.449 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054380+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.450 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645644+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.450 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059874+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.450 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.656499+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.450 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061665+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.451 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.165344+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.451 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057712+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.451 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160614+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.451 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.655228+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.451 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652396+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.452 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652639+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.452 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647082+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.452 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161648+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.452 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644788+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.453 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161366+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.453 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650536+0000 34'96 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.453 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057570+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.453 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062423+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.454 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061857+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.454 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651257+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.454 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.650322+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.454 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064099+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.454 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060925+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.455 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646360+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.455 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 1 0 0 601 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.655464+0000 53'96 56:122 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.455 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648831+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.455 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059519+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.456 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055783+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.456 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058878+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.456 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055308+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.456 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646801+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.456 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057260+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.457 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059379+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.457 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.056265+0000 32'95 55:140 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.457 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649448+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.457 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055108+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.458 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646439+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.458 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056459+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.458 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.059557+0000 34'96 55:148 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.458 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058386+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.459 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060541+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.460 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651884+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.460 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063303+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.460 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060893+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.461 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652020+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.461 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063243+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.461 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651093+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.461 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644530+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.461 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061408+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.462 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650813+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.462 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055246+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.462 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061302+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.462 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159501+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.463 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160463+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.463 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.654813+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.463 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063624+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.463 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652317+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.464 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062782+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.464 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651456+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.464 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159722+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.464 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648810+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.464 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652809+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.465 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651444+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.465 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054311+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.465 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160718+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.465 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650851+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.466 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058230+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.466 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651009+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.466 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.163179+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.466 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.592753+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.467 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064015+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.467 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647935+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.467 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064861+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.467 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.159494+0000 34'96 55:148 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.467 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652699+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.468 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.159523+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.468 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.062968+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.468 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647259+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.468 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.162249+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.469 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.056263+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.469 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645544+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.469 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.054889+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.469 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.161101+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.469 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644646+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.470 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.656444+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.470 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.058220+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.470 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.059923+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.470 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645471+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.471 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652814+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.471 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647813+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.471 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652308+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.471 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644586+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.471 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648452+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.472 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.646610+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.472 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.057203+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.472 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648029+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.472 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655913+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.473 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.060941+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.473 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655011+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.473 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.645795+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.473 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.054246+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.473 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058839+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.474 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.162128+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.474 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058635+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.474 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060058+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.474 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:14:00.475 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:14:00.475 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:14:00.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2021-07-26T06:14:00.476 INFO:tasks.workunit.client.0.smithi094.stdout:wait for peering 2021-07-26T06:14:00.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:14:00.813 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:14:00.814 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.656870+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.814 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061855+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.814 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.056238+0000 34'96 55:137 [0,2] 0 [0,2] 0 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.814 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059226+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.815 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 1 0 0 617 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651961+0000 32'95 56:137 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.815 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055365+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.815 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 1 0 0 617 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650564+0000 34'96 56:149 [2,0] 2 [2,0] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.816 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 3 0 0 1851 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651681+0000 32'95 56:137 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.816 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161682+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.816 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056354+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.816 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063687+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.817 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060493+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.817 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647254+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.817 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649803+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.818 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650413+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.818 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064477+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.818 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.648792+0000 32'95 56:137 [2,0] 2 [2,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:14:00.819 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649585+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.819 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061742+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.819 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648454+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.819 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649644+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.820 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645510+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.820 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:36.981078+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.820 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054380+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.821 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.645644+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.821 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059874+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.821 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.656499+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.821 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061665+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.821 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.165344+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.822 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057712+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.822 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160614+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.822 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.655228+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.822 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652396+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.823 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652639+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.823 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647082+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.823 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161648+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.824 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644788+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.824 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.161366+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.824 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.650536+0000 34'96 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.825 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057570+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.825 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062423+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.825 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061857+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.825 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651257+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.825 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.650322+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.826 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064099+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.826 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060925+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.826 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646360+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.827 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 1 0 0 601 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.655464+0000 53'96 56:122 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.827 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648831+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.827 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059519+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.828 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055783+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.828 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058878+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.828 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055308+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.829 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646801+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.829 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.057260+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.829 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.059379+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.829 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+undersized 2021-07-26T06:12:37.056265+0000 32'95 55:140 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.830 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.649448+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.830 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055108+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.830 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.646439+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.830 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.056459+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.831 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.059557+0000 34'96 55:148 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.831 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058386+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.831 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060541+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.831 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651884+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.832 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063303+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.832 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.060893+0000 32'95 55:125 [0,2] 0 [0,2] 0 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.832 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652020+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.833 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063243+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.833 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651093+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.833 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.644530+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.834 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061408+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.834 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650813+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.834 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.055246+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.835 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.061302+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.835 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159501+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.835 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160463+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.835 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.654813+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.063624+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652317+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.062782+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651456+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.159722+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.648810+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652809+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651444+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.054311+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.160718+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.650851+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.058230+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.651009+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.163179+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.592753+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064015+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.647935+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.064861+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 2 0 0 1075 0 0 18 18 active+undersized+degraded 2021-07-26T06:12:37.159494+0000 34'96 55:148 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 1 0 0 458 0 0 17 17 active+undersized+degraded 2021-07-26T06:12:37.652699+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.159523+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.062968+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647259+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.162249+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.056263+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645544+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.054889+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.161101+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644646+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.656444+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.058220+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.059923+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.645471+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652814+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.647813+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.845 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.652308+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.845 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.644586+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.846 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648452+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.846 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.646610+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.846 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.057203+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.648029+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655913+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.060941+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+undersized 2021-07-26T06:12:37.655011+0000 32'95 56:133 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.645795+0000 32'95 56:121 [2,0] 2 [2,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.054246+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058839+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.162128+0000 32'95 55:125 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.058635+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+undersized 2021-07-26T06:12:37.060058+0000 32'95 55:136 [0,2] 0 [0,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:00.850 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:14:00.850 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:14:00.851 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:14:00.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2021-07-26T06:14:13.295 INFO:tasks.workunit.client.0.smithi094.stdout:killing divergent 1 2021-07-26T06:14:13.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2021-07-26T06:14:13.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:14:13.653 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:14:13.653 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.959372+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.653 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852178+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.654 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 0 0 0 1075 0 0 18 18 active+clean+wait 2021-07-26T06:13:59.837992+0000 34'96 60:142 [1,0,2] 1 [1,0,2] 1 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.654 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848880+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.654 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:13:59.336264+0000 32'95 59:149 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.654 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.842483+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.655 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 0 0 0 617 0 0 18 18 active+clean 2021-07-26T06:13:59.689114+0000 34'96 60:161 [2,0,1] 2 [2,0,1] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.655 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 0 0 0 1851 0 0 17 17 active+clean 2021-07-26T06:13:59.578050+0000 32'95 60:149 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.655 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.981841+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.655 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.845438+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.655 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852664+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.656 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851778+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.656 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.856286+0000 32'95 60:147 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.656 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.694523+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:16.865733+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.656 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.336016+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.657 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.856226+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:20.682227+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.657 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:13:59.688882+0000 32'95 60:149 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:14:13.657 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332204+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:18.872678+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.657 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.980306+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.658 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850155+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.658 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.857585+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.658 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.330188+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.658 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.747563+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.659 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.843103+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.659 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332316+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.659 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855732+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.659 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689844+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:17.913907+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.659 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851977+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.660 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.958303+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.660 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849946+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.660 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858685+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:18.672932+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.660 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.957864+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.661 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.853772+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.661 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.960696+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.661 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.846622+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.661 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858994+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.661 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:13:59.335043+0000 32'95 59:150 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:13:59.334950+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.662 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.854166+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.662 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 0 0 0 1075 0 0 18 18 active+clean+wait 2021-07-26T06:13:59.857828+0000 34'96 60:147 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.662 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.844427+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.662 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849413+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.663 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.846140+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.663 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.333099+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.663 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.332447+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.663 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.855002+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.664 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849095+0000 32'95 60:140 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.664 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.687652+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.664 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 1 1 0 0 601 0 0 18 18 active+recovery_wait+degraded+wait 2021-07-26T06:13:59.854537+0000 53'96 60:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.664 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.694189+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.664 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851365+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:12.806402+0000 32'95 2021-07-26T06:12:12.806402+0000 0 2021-07-26T06:14:13.665 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+recovery_wait+degraded 2021-07-26T06:13:59.843607+0000 32'95 60:147 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.665 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.845789+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.665 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849707+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.665 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.693353+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.666 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855572+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.666 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.845800+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:14.633991+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.666 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:13:59.845609+0000 32'95 60:152 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.666 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.334577+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.667 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848222+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.667 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.856043+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.667 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+recovery_wait+degraded 2021-07-26T06:13:59.844131+0000 32'95 60:147 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.667 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:13:59.845815+0000 34'96 60:160 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.668 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.849715+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.668 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851788+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.668 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848931+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.668 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855975+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.668 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850590+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.669 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848666+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.669 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852183+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:20.740167+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.669 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689358+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.669 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.836712+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.670 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850804+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.670 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.694934+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.670 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.836418+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.670 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851967+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.671 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.856539+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.671 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.846289+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.671 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.696190+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.671 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858417+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.671 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849180+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.672 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.846319+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.672 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850371+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.672 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.981169+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.672 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.331986+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.673 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689597+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.673 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.690475+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.673 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.842450+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.673 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.981509+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.673 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332696+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.674 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.844817+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.674 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858049+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.674 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.848109+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.674 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.697321+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.675 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.854039+0000 32'95 60:149 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.675 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.688592+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.675 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.856216+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.675 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:13:59.846002+0000 34'96 60:160 [0,1,2] 0 [0,1,2] 0 34'96 2021-07-26T06:12:17.759855+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.676 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.695365+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.676 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.854224+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:13.843874+0000 32'95 2021-07-26T06:12:13.843874+0000 0 2021-07-26T06:14:13.676 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852459+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.676 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.688184+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.676 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.854392+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.677 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.843587+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.677 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855810+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.677 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.839957+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.677 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858870+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.678 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332105+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.678 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 1 0 0 458 0 0 17 17 active+recovering+degraded 2021-07-26T06:13:59.698106+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.678 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851069+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:14.812154+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.678 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851589+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.678 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.202335+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.679 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.336581+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.679 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.693749+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.679 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.859510+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.679 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.202136+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.680 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689957+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.680 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332638+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.680 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848437+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.680 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858164+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.680 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849634+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.681 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.845965+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.681 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.336983+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.681 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.847994+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.681 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.835626+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.682 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.851393+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.682 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.859260+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.682 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.845624+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.682 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.849925+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:13.682 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:14:13.683 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:14:13.684 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:14:13.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2021-07-26T06:14:13.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:14:13.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:14:13.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:14:13.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:14:13.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:14:13.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:14:13.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.18330.out 2021-07-26T06:14:13.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.18330.out 2021-07-26T06:14:13.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-07-26T06:14:13.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:14:13.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:14:13.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:14:13.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-07-26T06:14:13.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.18330.out 2021-07-26T06:14:14.754 INFO:tasks.workunit.client.0.smithi094.stderr:Exporting 1.1b info 1.1b( v 32'95 (31'78,32'95] local-lis/les=58/60 n=1 ec=33/23 lis/c=58/35 les/c/f=60/37/0 sis=58) 2021-07-26T06:14:14.763 INFO:tasks.workunit.client.0.smithi094.stderr:Read #1:d9cbd1d6:::existing_64:head# 2021-07-26T06:14:14.788 INFO:tasks.workunit.client.0.smithi094.stderr:Export successful 2021-07-26T06:14:14.794 INFO:tasks.workunit.client.0.smithi094.stdout: marking collection for removal 2021-07-26T06:14:14.795 INFO:tasks.workunit.client.0.smithi094.stdout:setting '_remove' omap key 2021-07-26T06:14:14.795 INFO:tasks.workunit.client.0.smithi094.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2021-07-26T06:14:15.029 INFO:tasks.workunit.client.0.smithi094.stderr:Remove successful 2021-07-26T06:14:15.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.18330.out 2021-07-26T06:14:15.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-07-26T06:14:15.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:14:15.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:14:15.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:14:15.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-07-26T06:14:15.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.18330.out 2021-07-26T06:14:16.304 INFO:tasks.workunit.client.0.smithi094.stdout:get_pg_num_history pg_num_history pg_num_history(e60 pg_nums {1={23=1,33=128}} deleted_pools ) 2021-07-26T06:14:16.304 INFO:tasks.workunit.client.0.smithi094.stdout:Importing pgid 1.1b 2021-07-26T06:14:16.304 INFO:tasks.workunit.client.0.smithi094.stdout:Write #1:d9cbd1d6:::existing_64:head# 2021-07-26T06:14:16.304 INFO:tasks.workunit.client.0.smithi094.stdout:snapset 0=[]:{} 2021-07-26T06:14:16.374 INFO:tasks.workunit.client.0.smithi094.stdout:write_pg epoch 60 info 1.1b( v 32'95 (31'78,32'95] local-lis/les=58/60 n=1 ec=33/23 lis/c=58/35 les/c/f=60/37/0 sis=58) 2021-07-26T06:14:16.449 INFO:tasks.workunit.client.0.smithi094.stdout:Import successful 2021-07-26T06:14:16.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2021-07-26T06:14:16.994 INFO:tasks.workunit.client.0.smithi094.stdout:reviving divergent 1 2021-07-26T06:14:16.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:14:17.349 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:14:17.349 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.959372+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.349 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852178+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 0 0 0 1075 0 0 18 18 active+clean+wait 2021-07-26T06:13:59.837992+0000 34'96 60:142 [1,0,2] 1 [1,0,2] 1 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848880+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:13:59.336264+0000 32'95 59:149 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.350 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.842483+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 0 0 0 617 0 0 18 18 active+clean 2021-07-26T06:13:59.689114+0000 34'96 60:161 [2,0,1] 2 [2,0,1] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 0 0 0 1851 0 0 17 17 active+clean 2021-07-26T06:13:59.578050+0000 32'95 60:149 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.981841+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.351 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.845438+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852664+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851778+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.856286+0000 32'95 60:147 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.694523+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:16.865733+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.352 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.336016+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.856226+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:20.682227+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:13:59.688882+0000 32'95 60:149 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:14:17.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332204+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:18.872678+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.980306+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.353 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850155+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.857585+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.330188+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.747563+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.354 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.843103+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332316+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855732+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689844+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:17.913907+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.355 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851977+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.958303+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849946+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858685+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:18.672932+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.957864+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.356 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.853772+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.960696+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.846622+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858994+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:13:59.335043+0000 32'95 59:150 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:13:59.334950+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.357 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.854166+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 0 0 0 1075 0 0 18 18 active+clean+wait 2021-07-26T06:13:59.857828+0000 34'96 60:147 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.844427+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849413+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.358 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.846140+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.333099+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.332447+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.855002+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849095+0000 32'95 60:140 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.359 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.687652+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 1 1 0 0 601 0 0 18 18 active+recovery_wait+degraded+wait 2021-07-26T06:13:59.854537+0000 53'96 60:145 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.694189+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851365+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:12.806402+0000 32'95 2021-07-26T06:12:12.806402+0000 0 2021-07-26T06:14:17.360 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 1 0 0 458 0 0 17 17 active+recovery_wait+degraded 2021-07-26T06:13:59.843607+0000 32'95 60:147 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.845789+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849707+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.693353+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855572+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.845800+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:14.633991+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.361 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:13:59.845609+0000 32'95 60:152 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.334577+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848222+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.856043+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.362 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 1 0 0 458 0 0 17 17 active+recovery_wait+degraded 2021-07-26T06:13:59.844131+0000 32'95 60:147 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:13:59.845815+0000 34'96 60:160 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.849715+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851788+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848931+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.363 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855975+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850590+0000 32'95 60:141 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848666+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852183+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:20.740167+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689358+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.364 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.836712+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850804+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.694934+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.836418+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851967+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.365 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.856539+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.846289+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.696190+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858417+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.366 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849180+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.846319+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.850371+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.981169+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.331986+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.367 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689597+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.690475+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.842450+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.981509+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332696+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.368 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.844817+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858049+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.848109+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.697321+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.369 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.854039+0000 32'95 60:149 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.688592+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.856216+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:13:59.846002+0000 34'96 60:160 [0,1,2] 0 [0,1,2] 0 34'96 2021-07-26T06:12:17.759855+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.695365+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.370 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.854224+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:13.843874+0000 32'95 2021-07-26T06:12:13.843874+0000 0 2021-07-26T06:14:17.371 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.852459+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.371 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.688184+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.371 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.854392+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.371 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.843587+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.371 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.855810+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.372 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.839957+0000 32'95 60:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.372 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858870+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.372 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332105+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.372 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 1 0 0 458 0 0 17 17 active+recovering+degraded 2021-07-26T06:13:59.698106+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.372 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851069+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:14.812154+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.373 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.851589+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.373 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.202335+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.373 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.336581+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.373 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.693749+0000 32'95 60:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.373 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.859510+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.374 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.202136+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.374 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.689957+0000 32'95 60:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.374 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.332638+0000 32'95 59:145 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.374 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.848437+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.374 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.858164+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.375 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.849634+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.375 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.845965+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.375 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:13:59.336983+0000 32'95 59:145 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.375 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.847994+0000 32'95 60:146 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.376 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.835626+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.376 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.851393+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.376 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+clean+wait 2021-07-26T06:13:59.859260+0000 32'95 60:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.376 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.845624+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.377 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:13:59.849925+0000 32'95 60:148 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:17.377 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:14:17.377 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:14:17.381 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:14:17.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-07-26T06:14:17.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:14:17.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:14:17.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:14:17.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:14:17.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:14:17.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:14:17.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:14:17.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:14:17.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:14:17.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:14:17.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:14:17.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:14:17.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:14:17.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:14:17.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:14:17.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:14:17.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:14:17.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:14:17.388 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:14:17.388 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:17.388 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:17.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:17.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:17.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:14:17.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:14:17.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:14:17.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:14:17.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:14:17.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:14:17.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:14:17.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:14:17.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:14:17.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:14:17.391 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:14:17.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:14:17.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:14:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:14:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:14:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:14:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:14:17.421 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:17.416+0000 7fa2e00aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:17.424 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:17.420+0000 7fa2e00aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:17.427 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:17.420+0000 7fa2e00aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:17.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:14:17.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:14:17.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:14:17.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:14:17.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:14:17.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:14:17.726 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:14:17.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:14:17.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:14:17.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:14:17.970 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:17.964+0000 7fa2e00aaf40 -1 Falling back to public interface 2021-07-26T06:14:18.127 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 39 last_clean_interval [10,37) [v2:127.0.0.1:6818/55407,v1:127.0.0.1:6819/55407] [v2:127.0.0.1:6820/55407,v1:127.0.0.1:6821/55407] exists,up 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c 2021-07-26T06:14:18.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:14:18.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:14:18.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:14:18.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2021-07-26T06:14:18.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:14:18.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:14:18.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:14:18.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:14:18.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:14:18.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:14:18.131 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:14:18.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:14:18.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:14:18.535 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 39 last_clean_interval [10,37) [v2:127.0.0.1:6818/55407,v1:127.0.0.1:6819/55407] [v2:127.0.0.1:6820/55407,v1:127.0.0.1:6821/55407] exists,up 64b2a2e5-8ab4-4197-9d59-4b39f363ac4c 2021-07-26T06:14:18.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:14:18.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:14:18.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:14:18.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2021-07-26T06:14:21.043 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:21.036+0000 7fa2e00aaf40 -1 osd.1 60 log_to_monitors {default=true} 2021-07-26T06:14:38.538 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2021-07-26T06:14:38.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:14:38.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:14:38.540 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:14:38.540 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:38.540 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:38.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:14:38.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2021-07-26T06:14:38.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.1.asok dump_ops_in_flight 2021-07-26T06:14:38.630 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:14:38.630 INFO:tasks.workunit.client.0.smithi094.stdout: "ops": [], 2021-07-26T06:14:38.631 INFO:tasks.workunit.client.0.smithi094.stdout: "num_ops": 0 2021-07-26T06:14:38.631 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:14:38.637 INFO:tasks.workunit.client.0.smithi094.stdout:allowing recovery 2021-07-26T06:14:38.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2021-07-26T06:14:38.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:14:38.999 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:14:38.999 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.556810+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.000 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497558+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.000 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.332443+0000 34'96 65:153 [1,0,2] 1 [1,0,2] 1 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.000 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335927+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.000 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:14:23.347953+0000 32'95 65:164 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.001 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323184+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.001 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 0 0 0 617 0 0 18 18 active+clean 2021-07-26T06:14:23.496732+0000 34'96 65:176 [2,0,1] 2 [2,0,1] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.001 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 0 0 0 1851 0 0 17 17 active+clean 2021-07-26T06:14:23.782364+0000 32'95 65:164 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.001 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.701299+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.002 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.493728+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.002 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.495715+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.002 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.496975+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.002 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334850+0000 32'95 65:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.003 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:29.545416+0000 32'95 65:165 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:14:29.545267+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.003 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335313+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.003 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.429692+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:20.682227+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.003 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:23.331987+0000 32'95 65:164 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:14:39.004 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333296+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:18.872678+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.004 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.498072+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.004 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333135+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.004 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.430980+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.005 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331537+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.005 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.316703+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.006 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:32.418862+0000 32'95 65:167 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:14:32.418685+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.006 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.324322+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.007 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.329654+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.007 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.531431+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:17.913907+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.007 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.779754+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.007 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.501216+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.008 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332445+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.008 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335714+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:18.672932+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.008 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.427574+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.008 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.558164+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.009 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.559111+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.009 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.330337+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.009 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.337058+0000 60'96 65:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.009 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:23.318414+0000 32'95 65:165 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:13:59.334950+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.010 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.558651+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.010 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.432362+0000 34'96 65:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.010 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.326073+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.010 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.556043+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.010 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332169+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.011 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335532+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.011 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.333550+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.011 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.496578+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.011 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.426449+0000 32'95 65:151 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.012 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334788+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.012 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 0 0 0 601 0 0 18 18 active+clean 2021-07-26T06:14:24.302497+0000 53'96 65:161 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.012 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:14:25.407932+0000 32'95 65:165 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:14:25.407757+0000 32'95 2021-07-26T06:14:25.407757+0000 0 2021-07-26T06:14:39.013 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.778711+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:12.806402+0000 32'95 2021-07-26T06:12:12.806402+0000 0 2021-07-26T06:14:39.013 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.325706+0000 32'95 65:161 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.013 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331734+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.013 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:34.183432+0000 32'95 65:151 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:14:34.183258+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.014 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.349254+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.014 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332868+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.014 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.330131+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:14.633991+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.014 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:23.326368+0000 32'95 65:166 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.015 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334931+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.015 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333259+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.015 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334625+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.015 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.325940+0000 32'95 65:161 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.016 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.494162+0000 34'96 65:174 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.016 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497558+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.016 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.779244+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.016 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.554833+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.016 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335569+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.017 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.428785+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.017 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335305+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.017 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.780842+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:20.740167+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.017 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497686+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.017 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 2 0 0 0 0 1234 0 0 17 17 active+clean 2021-07-26T06:14:29.116643+0000 32'95 65:162 [1,2,0] 1 [1,2,0] 1 32'95 2021-07-26T06:14:29.116455+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.018 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.429543+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.018 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.351688+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.018 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323407+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.018 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497288+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.019 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.498576+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.019 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.494604+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.019 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.352691+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.019 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335274+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.020 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.555697+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.020 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332382+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.020 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.427472+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.020 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.700045+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.020 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333097+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.021 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.531228+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.021 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.348495+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.021 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.320515+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.021 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.700744+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.022 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333767+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.022 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.326237+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.022 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.432678+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.023 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.494800+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.023 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.228054+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.023 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.781552+0000 32'95 65:163 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.023 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331777+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.023 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497960+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.024 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.494407+0000 34'96 65:174 [0,1,2] 0 [0,1,2] 0 34'96 2021-07-26T06:12:17.759855+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.024 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.351980+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.024 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.782238+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:13.843874+0000 32'95 2021-07-26T06:12:13.843874+0000 0 2021-07-26T06:14:39.024 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497827+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.025 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334981+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.025 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.558810+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.025 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.319974+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.025 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333081+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.026 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.325446+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.026 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.557294+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.026 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323805+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.026 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.353311+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.027 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323920+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:14.812154+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.027 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.492124+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.027 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334582+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.027 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.500263+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.028 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.349527+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.028 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.780408+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.028 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331316+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.028 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.348162+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.029 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.324962+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.029 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334890+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.030 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335048+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.030 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.556463+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.031 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331949+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.031 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.500415+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.031 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.324187+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.032 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.322865+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.032 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.491296+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.033 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.557759+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.033 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.493948+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.033 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.497827+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.034 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:14:39.034 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:14:39.035 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:14:39.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-07-26T06:14:39.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2021-07-26T06:14:39.150 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:14:39.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-07-26T06:14:39.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2021-07-26T06:14:39.302 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:14:39.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-07-26T06:14:39.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2021-07-26T06:14:39.459 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:14:39.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2021-07-26T06:14:39.467 INFO:tasks.workunit.client.0.smithi094.stdout:reading divergent objects 2021-07-26T06:14:39.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2021-07-26T06:14:39.835 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:14:39.835 INFO:tasks.workunit.client.0.smithi094.stdout:1.7f 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.556810+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.835 INFO:tasks.workunit.client.0.smithi094.stdout:1.39 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497558+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.38 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.332443+0000 34'96 65:153 [1,0,2] 1 [1,0,2] 1 34'96 2021-07-26T06:12:03.500571+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.37 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335927+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:09.475143+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.36 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:14:23.347953+0000 32'95 65:164 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:02.953580+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.35 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323184+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.836 INFO:tasks.workunit.client.0.smithi094.stdout:1.34 1 0 0 0 0 617 0 0 18 18 active+clean 2021-07-26T06:14:23.496732+0000 34'96 65:176 [2,0,1] 2 [2,0,1] 2 34'96 2021-07-26T06:12:05.981341+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.33 3 0 0 0 0 1851 0 0 17 17 active+clean 2021-07-26T06:14:23.782364+0000 32'95 65:164 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:07.947957+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.32 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.701299+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.31 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.493728+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.837 INFO:tasks.workunit.client.0.smithi094.stdout:1.30 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.495715+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.2f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.496975+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.2e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334850+0000 32'95 65:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.2d 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:29.545416+0000 32'95 65:165 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:14:29.545267+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.838 INFO:tasks.workunit.client.0.smithi094.stdout:1.2c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335313+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.2b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.429692+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:20.682227+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.2a 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:23.331987+0000 32'95 65:164 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:06.933890+0000 32'95 2021-07-26T06:12:06.933890+0000 0 2021-07-26T06:14:39.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.29 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333296+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 32'95 2021-07-26T06:12:18.872678+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.28 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.498072+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.839 INFO:tasks.workunit.client.0.smithi094.stdout:1.27 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333135+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.26 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.430980+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.25 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331537+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.24 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.316703+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.23 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:32.418862+0000 32'95 65:167 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:14:32.418685+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.840 INFO:tasks.workunit.client.0.smithi094.stdout:1.22 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.324322+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.21 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.329654+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.20 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.531431+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:12:17.913907+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.1f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.779754+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.1e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.501216+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.841 INFO:tasks.workunit.client.0.smithi094.stdout:1.1d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332445+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335714+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:18.672932+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.427574+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.9 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.558164+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.842 INFO:tasks.workunit.client.0.smithi094.stdout:1.8 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.559111+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.7 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.330337+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.6 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.337058+0000 60'96 65:148 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.5 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:23.318414+0000 32'95 65:165 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:13:59.334950+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.4 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.558651+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.843 INFO:tasks.workunit.client.0.smithi094.stdout:1.3 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.432362+0000 34'96 65:158 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.2 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.326073+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.556043+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.3a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332169+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.1 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335532+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.844 INFO:tasks.workunit.client.0.smithi094.stdout:1.7e 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.333550+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.845 INFO:tasks.workunit.client.0.smithi094.stdout:1.1c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.496578+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.845 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.426449+0000 32'95 65:151 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:11:59.499155+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.845 INFO:tasks.workunit.client.0.smithi094.stdout:1.d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334788+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.845 INFO:tasks.workunit.client.0.smithi094.stdout:1.e 1 0 0 0 0 601 0 0 18 18 active+clean 2021-07-26T06:14:24.302497+0000 53'96 65:161 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.845 INFO:tasks.workunit.client.0.smithi094.stdout:1.f 1 0 0 0 0 617 0 0 17 17 active+clean 2021-07-26T06:14:25.407932+0000 32'95 65:165 [2,0,1] 2 [2,0,1] 2 32'95 2021-07-26T06:14:25.407757+0000 32'95 2021-07-26T06:14:25.407757+0000 0 2021-07-26T06:14:39.846 INFO:tasks.workunit.client.0.smithi094.stdout:1.10 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.778711+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:12.806402+0000 32'95 2021-07-26T06:12:12.806402+0000 0 2021-07-26T06:14:39.846 INFO:tasks.workunit.client.0.smithi094.stdout:1.11 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.325706+0000 32'95 65:161 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.846 INFO:tasks.workunit.client.0.smithi094.stdout:1.12 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331734+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.846 INFO:tasks.workunit.client.0.smithi094.stdout:1.13 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:34.183432+0000 32'95 65:151 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:14:34.183258+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.14 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.349254+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.15 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332868+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.16 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.330131+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:14.633991+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.17 0 0 0 0 0 0 0 0 17 17 active+clean 2021-07-26T06:14:23.326368+0000 32'95 65:166 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:07.815127+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.847 INFO:tasks.workunit.client.0.smithi094.stdout:1.18 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334931+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.19 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333259+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.1a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334625+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.1b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.325940+0000 32'95 65:161 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.3b 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.494162+0000 34'96 65:174 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.848 INFO:tasks.workunit.client.0.smithi094.stdout:1.3c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497558+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.3d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.779244+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.3e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.554833+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.3f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335569+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:07.454916+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.40 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.428785+0000 32'95 65:152 [1,0,2] 1 [1,0,2] 1 32'95 2021-07-26T06:12:06.434443+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.849 INFO:tasks.workunit.client.0.smithi094.stdout:1.41 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335305+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.850 INFO:tasks.workunit.client.0.smithi094.stdout:1.42 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.780842+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:20.740167+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.850 INFO:tasks.workunit.client.0.smithi094.stdout:1.43 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497686+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.850 INFO:tasks.workunit.client.0.smithi094.stdout:1.44 2 0 0 0 0 1234 0 0 17 17 active+clean 2021-07-26T06:14:29.116643+0000 32'95 65:162 [1,2,0] 1 [1,2,0] 1 32'95 2021-07-26T06:14:29.116455+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.850 INFO:tasks.workunit.client.0.smithi094.stdout:1.45 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.429543+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.851 INFO:tasks.workunit.client.0.smithi094.stdout:1.46 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.351688+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.851 INFO:tasks.workunit.client.0.smithi094.stdout:1.47 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323407+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.851 INFO:tasks.workunit.client.0.smithi094.stdout:1.48 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497288+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.851 INFO:tasks.workunit.client.0.smithi094.stdout:1.49 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.498576+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.852 INFO:tasks.workunit.client.0.smithi094.stdout:1.4a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.494604+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.852 INFO:tasks.workunit.client.0.smithi094.stdout:1.4b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.352691+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.852 INFO:tasks.workunit.client.0.smithi094.stdout:1.4c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335274+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.852 INFO:tasks.workunit.client.0.smithi094.stdout:1.4d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.555697+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.852 INFO:tasks.workunit.client.0.smithi094.stdout:1.4e 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.332382+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.853 INFO:tasks.workunit.client.0.smithi094.stdout:1.4f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.427472+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.853 INFO:tasks.workunit.client.0.smithi094.stdout:1.50 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.700045+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.853 INFO:tasks.workunit.client.0.smithi094.stdout:1.51 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333097+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.853 INFO:tasks.workunit.client.0.smithi094.stdout:1.52 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.531228+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.854 INFO:tasks.workunit.client.0.smithi094.stdout:1.53 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.348495+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.854 INFO:tasks.workunit.client.0.smithi094.stdout:1.54 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.320515+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.854 INFO:tasks.workunit.client.0.smithi094.stdout:1.55 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.700744+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.854 INFO:tasks.workunit.client.0.smithi094.stdout:1.56 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333767+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.854 INFO:tasks.workunit.client.0.smithi094.stdout:1.57 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.326237+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.855 INFO:tasks.workunit.client.0.smithi094.stdout:1.58 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.432678+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.855 INFO:tasks.workunit.client.0.smithi094.stdout:1.59 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.494800+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.855 INFO:tasks.workunit.client.0.smithi094.stdout:1.5a 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.228054+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.855 INFO:tasks.workunit.client.0.smithi094.stdout:1.5b 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.781552+0000 32'95 65:163 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.856 INFO:tasks.workunit.client.0.smithi094.stdout:1.5c 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331777+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.856 INFO:tasks.workunit.client.0.smithi094.stdout:1.5d 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497960+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.856 INFO:tasks.workunit.client.0.smithi094.stdout:1.5e 2 0 0 0 0 1075 0 0 18 18 active+clean 2021-07-26T06:14:23.494407+0000 34'96 65:174 [0,1,2] 0 [0,1,2] 0 34'96 2021-07-26T06:12:17.759855+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.856 INFO:tasks.workunit.client.0.smithi094.stdout:1.5f 1 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.351980+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.856 INFO:tasks.workunit.client.0.smithi094.stdout:1.60 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.782238+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 32'95 2021-07-26T06:12:13.843874+0000 32'95 2021-07-26T06:12:13.843874+0000 0 2021-07-26T06:14:39.857 INFO:tasks.workunit.client.0.smithi094.stdout:1.61 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.497827+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.857 INFO:tasks.workunit.client.0.smithi094.stdout:1.62 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334981+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.857 INFO:tasks.workunit.client.0.smithi094.stdout:1.63 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.558810+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.857 INFO:tasks.workunit.client.0.smithi094.stdout:1.64 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.319974+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.857 INFO:tasks.workunit.client.0.smithi094.stdout:1.65 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.333081+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.858 INFO:tasks.workunit.client.0.smithi094.stdout:1.66 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.325446+0000 32'95 65:162 [0,1,2] 0 [0,1,2] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.858 INFO:tasks.workunit.client.0.smithi094.stdout:1.67 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.557294+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.858 INFO:tasks.workunit.client.0.smithi094.stdout:1.68 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323805+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.858 INFO:tasks.workunit.client.0.smithi094.stdout:1.69 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.353311+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.859 INFO:tasks.workunit.client.0.smithi094.stdout:1.6a 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.323920+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 32'95 2021-07-26T06:12:14.812154+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.859 INFO:tasks.workunit.client.0.smithi094.stdout:1.6b 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.492124+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.859 INFO:tasks.workunit.client.0.smithi094.stdout:1.6c 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334582+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.860 INFO:tasks.workunit.client.0.smithi094.stdout:1.6d 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.500263+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.860 INFO:tasks.workunit.client.0.smithi094.stdout:1.6e 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.349527+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.860 INFO:tasks.workunit.client.0.smithi094.stdout:1.6f 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.780408+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.860 INFO:tasks.workunit.client.0.smithi094.stdout:1.70 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331316+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.861 INFO:tasks.workunit.client.0.smithi094.stdout:1.71 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.348162+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.861 INFO:tasks.workunit.client.0.smithi094.stdout:1.72 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.324962+0000 32'95 65:160 [2,0,1] 2 [2,0,1] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.861 INFO:tasks.workunit.client.0.smithi094.stdout:1.73 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.334890+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.861 INFO:tasks.workunit.client.0.smithi094.stdout:1.74 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.335048+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.862 INFO:tasks.workunit.client.0.smithi094.stdout:1.75 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.556463+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.862 INFO:tasks.workunit.client.0.smithi094.stdout:1.76 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.331949+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.862 INFO:tasks.workunit.client.0.smithi094.stdout:1.77 0 0 0 0 0 458 0 0 17 17 active+clean 2021-07-26T06:14:23.500415+0000 32'95 65:160 [2,1,0] 2 [2,1,0] 2 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.862 INFO:tasks.workunit.client.0.smithi094.stdout:1.78 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.324187+0000 32'95 65:157 [1,2,0] 1 [1,2,0] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.863 INFO:tasks.workunit.client.0.smithi094.stdout:1.79 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.322865+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.863 INFO:tasks.workunit.client.0.smithi094.stdout:1.7a 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.491296+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.863 INFO:tasks.workunit.client.0.smithi094.stdout:1.7b 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.557759+0000 32'95 65:146 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.863 INFO:tasks.workunit.client.0.smithi094.stdout:1.7c 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.493948+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.863 INFO:tasks.workunit.client.0.smithi094.stdout:1.7d 0 0 0 0 0 457 0 0 17 17 active+clean 2021-07-26T06:14:23.497827+0000 32'95 65:162 [0,2,1] 0 [0,2,1] 0 0'0 2021-07-26T06:11:28.761830+0000 0'0 2021-07-26T06:11:28.761830+0000 0 2021-07-26T06:14:39.864 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:14:39.864 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:14:39.865 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:14:39.865 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2021-07-26T06:14:39.866 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2021-07-26T06:14:39.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:14:39.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2021-07-26T06:14:39.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:14:39.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2021-07-26T06:14:39.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:14:39.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2021-07-26T06:14:40.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:14:40.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2021-07-26T06:14:40.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2021-07-26T06:14:40.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:14:40.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2021-07-26T06:14:40.171 INFO:tasks.workunit.client.0.smithi094.stderr:error getting test/newobject_1: (2) No such file or directory 2021-07-26T06:14:40.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-07-26T06:14:40.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2021-07-26T06:14:40.219 INFO:tasks.workunit.client.0.smithi094.stderr:error getting test/newobject_2: (2) No such file or directory 2021-07-26T06:14:40.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2021-07-26T06:14:40.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-07-26T06:14:40.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:14:40.252 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.568+0000 7f64ed2b3700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 37'96 (31'4) delete 1:70b3bf12:::existing_4:head by client.4575.0:1 2021-07-26T06:12:10.548583+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.253 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.568+0000 7f64ed2b3700 20 _merge_object_divergent_entries: keeping 37'96 (31'4) delete 1:70b3bf12:::existing_4:head by client.4575.0:1 2021-07-26T06:12:10.548583+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.253 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.568+0000 7f64ed2b3700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.253 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.568+0000 7f64ed2b3700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.253 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.568+0000 7f64ed2b3700 10 _merge_object_divergent_entries: more recent entry found: 53'96 (31'4) modify 1:70b3bf12:::existing_4:head by client.4692.0:1 2021-07-26T06:13:13.391915+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0, already merged 2021-07-26T06:14:40.254 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.572+0000 7f64eeab6700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 37'96 (0'0) modify 1:75ac5091:::newobject_2:head by client.4567.0:1 2021-07-26T06:12:10.539381+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.254 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.572+0000 7f64eeab6700 20 _merge_object_divergent_entries: keeping 37'96 (0'0) modify 1:75ac5091:::newobject_2:head by client.4567.0:1 2021-07-26T06:12:10.539381+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.254 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.572+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.254 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.572+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.254 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.572+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-07-26T06:14:40.255 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:58.572+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-07-26T06:14:40.255 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 37'96 (31'2) modify 1:97db2a0f:::existing_2:head by client.4563.0:1 2021-07-26T06:12:10.534612+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.255 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 20 _merge_object_divergent_entries: keeping 37'96 (31'2) modify 1:97db2a0f:::existing_2:head by client.4563.0:1 2021-07-26T06:12:10.534612+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.255 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.256 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.256 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-07-26T06:14:40.256 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:14:40.256 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 37'96 2021-07-26T06:14:40.257 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 37'96 (31'2) modify 1:97db2a0f:::existing_2:head by client.4563.0:1 2021-07-26T06:12:10.534612+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.257 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-07-26T06:14:40.257 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.048+0000 7f64ef2b7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'78 2021-07-26T06:14:40.257 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64eeab6700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 37'96 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4573.0:1 2021-07-26T06:12:10.545656+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.258 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64eeab6700 20 _merge_object_divergent_entries: keeping 37'96 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4573.0:1 2021-07-26T06:12:10.545656+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.258 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.258 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.258 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-07-26T06:14:40.259 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64eeab6700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-07-26T06:14:40.259 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 37'96 (31'1) modify 1:d858c035:::existing_1:head by client.4562.0:1 2021-07-26T06:12:10.532389+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.259 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 20 _merge_object_divergent_entries: keeping 37'96 (31'1) modify 1:d858c035:::existing_1:head by client.4562.0:1 2021-07-26T06:12:10.532389+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.259 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.259 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.260 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-07-26T06:14:40.260 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:14:40.260 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 37'96 2021-07-26T06:14:40.279 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 37'96 (31'1) modify 1:d858c035:::existing_1:head by client.4562.0:1 2021-07-26T06:12:10.532389+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.280 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-07-26T06:14:40.280 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'78 2021-07-26T06:14:40.280 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 37'96 (31'3) delete 1:8881e6d8:::existing_3:head by client.4572.0:1 2021-07-26T06:12:10.545163+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.280 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 20 _merge_object_divergent_entries: keeping 37'96 (31'3) delete 1:8881e6d8:::existing_3:head by client.4572.0:1 2021-07-26T06:12:10.545163+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.280 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-07-26T06:14:40.281 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.281 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-07-26T06:14:40.281 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:14:40.281 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 37'96 2021-07-26T06:14:40.282 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 37'96 (31'3) delete 1:8881e6d8:::existing_3:head by client.4572.0:1 2021-07-26T06:12:10.545163+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.282 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-07-26T06:14:40.282 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:13:59.332+0000 7f64ee2b5700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'78 2021-07-26T06:14:40.282 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 37'96 (31'2) modify 1:97db2a0f:::existing_2:head by client.4563.0:1 2021-07-26T06:12:10.534612+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.283 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 20 _merge_object_divergent_entries: keeping 37'96 (31'2) modify 1:97db2a0f:::existing_2:head by client.4563.0:1 2021-07-26T06:12:10.534612+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.283 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.283 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.283 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-07-26T06:14:40.311 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:14:40.311 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 37'96 2021-07-26T06:14:40.311 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 37'96 (31'2) modify 1:97db2a0f:::existing_2:head by client.4563.0:1 2021-07-26T06:12:10.534612+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.311 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-07-26T06:14:40.312 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.2.log:2021-07-26T06:13:58.564+0000 7f4b00c6e700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'78 2021-07-26T06:14:40.312 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e9872f700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 37'96 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4573.0:1 2021-07-26T06:12:10.545656+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.312 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e9872f700 20 _merge_object_divergent_entries: keeping 37'96 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4573.0:1 2021-07-26T06:12:10.545656+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.312 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e9872f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.312 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e9872f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.313 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e9872f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-07-26T06:14:40.313 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e9872f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-07-26T06:14:40.313 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 37'96 (31'1) modify 1:d858c035:::existing_1:head by client.4562.0:1 2021-07-26T06:12:10.532389+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.313 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 20 _merge_object_divergent_entries: keeping 37'96 (31'1) modify 1:d858c035:::existing_1:head by client.4562.0:1 2021-07-26T06:12:10.532389+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.313 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:14:40.314 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.314 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-07-26T06:14:40.314 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:14:40.314 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 37'96 2021-07-26T06:14:40.319 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 37'96 (31'1) modify 1:d858c035:::existing_1:head by client.4562.0:1 2021-07-26T06:12:10.532389+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:14:40.319 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-07-26T06:14:40.319 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'78 2021-07-26T06:14:40.319 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 37'96 (31'3) delete 1:8881e6d8:::existing_3:head by client.4572.0:1 2021-07-26T06:12:10.545163+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.320 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 20 _merge_object_divergent_entries: keeping 37'96 (31'3) delete 1:8881e6d8:::existing_3:head by client.4572.0:1 2021-07-26T06:12:10.545163+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.320 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-07-26T06:14:40.320 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 37'96 last_divergent_update: 37'96 2021-07-26T06:14:40.320 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-07-26T06:14:40.320 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:14:40.321 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 37'96 2021-07-26T06:14:40.321 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 37'96 (31'3) delete 1:8881e6d8:::existing_3:head by client.4572.0:1 2021-07-26T06:12:10.545163+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2021-07-26T06:14:40.321 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-07-26T06:14:40.321 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.0.log:2021-07-26T06:13:58.568+0000 7f2e97f2e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'78 2021-07-26T06:14:40.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-07-26T06:14:40.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:14:40.341 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-07-26T06:14:40.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:14:40.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2021-07-26T06:14:40.365 INFO:tasks.workunit.client.0.smithi094.stdout:success 2021-07-26T06:14:40.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.18330.out 2021-07-26T06:14:40.366 INFO:tasks.workunit.client.0.smithi094.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.18330.out': No such file or directory 2021-07-26T06:14:40.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2021-07-26T06:14:40.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T06:14:40.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T06:14:40.873 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T06:14:40.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2021-07-26T06:14:40.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:14:40.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:14:40.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:14:40.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:14:40.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:14:41.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:14:41.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-07-26T06:14:41.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:14:41.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:14:41.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:14:41.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:14:41.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:14:41.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:14:41.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:14:41.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:14:41.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:14:41.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:14:41.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:14:41.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:14:41.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:14:41.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:14:41.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:14:41.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:14:41.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:14:41.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:14:41.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:14:41.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:14:41.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:14:41.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:14:41.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:14:41.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:41.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:41.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:14:41.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:14:41.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:14:41.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-07-26T06:14:41.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-07-26T06:14:41.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-07-26T06:14:41.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-07-26T06:14:41.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:14:41.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:14:41.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:14:41.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:14:41.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:14:41.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:14:41.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:14:41.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:14:41.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:14:41.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:14:41.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:14:41.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:14:41.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:14:41.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:14:41.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:14:41.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:14:41.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:14:41.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:14:41.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:14:41.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:14:41.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:14:41.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:14:41.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:14:41.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:41.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:41.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:14:41.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:14:41.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:14:41.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-07-26T06:14:41.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:14:41.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:41.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:41.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.18330 2021-07-26T06:14:41.101 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:14:41.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:14:41.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:14:41.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-07-26T06:14:41.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2021-07-26T06:14:41.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2021-07-26T06:14:41.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2021-07-26T06:14:41.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2021-07-26T06:14:41.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2021-07-26T06:14:41.104 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2021-07-26T06:14:41.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2021-07-26T06:14:41.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2021-07-26T06:14:41.106 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:14:41.107 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:14:41.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2021-07-26T06:14:41.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-07-26T06:14:41.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:14:41.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:14:41.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:14:41.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-07-26T06:14:41.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-07-26T06:14:41.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:14:41.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:14:41.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:14:41.548 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:14:41.549 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:41.549 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:41.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:41.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:14:41.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:14:41.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:14:41.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:14:41.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:14:41.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:14:41.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:14:41.791 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:14:41.792 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:14:41.792 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:14:41.792 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:14:41.792 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:41.793 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:41.793 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:14:41.794 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:14:41.794 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get fsid 2021-07-26T06:14:41.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:14:41.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:14:41.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:14:41.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:14:41.882 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:14:41.882 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:14:41.882 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:14:41.882 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:14:41.883 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:14:41.883 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:41.883 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:41.884 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-mon.a.asok 2021-07-26T06:14:41.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:14:41.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.18330/ceph-mon.a.asok config get mon_host 2021-07-26T06:14:41.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2021-07-26T06:14:41.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2021-07-26T06:14:41.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:14:41.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:14:41.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:14:41.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2021-07-26T06:14:41.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:14:42.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:14:42.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:14:42.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:14:42.875 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:14:42.875 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:42.876 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:42.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:42.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:14:42.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:14:42.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-07-26T06:14:42.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2021-07-26T06:14:42.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:14:42.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:14:42.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:14:42.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:14:42.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:14:42.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:14:42.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:14:42.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:14:42.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:14:42.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:14:42.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:14:42.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:14:42.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:14:42.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:14:42.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:14:42.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:14:42.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:14:42.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:14:42.918 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:14:42.919 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:42.919 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:42.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:42.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:42.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:14:42.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:14:42.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:14:42.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:14:42.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:14:42.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:14:42.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:14:42.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:14:42.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:14:42.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:14:42.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:14:42.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:14:42.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a83a0bc5-6fa4-4572-8fce-7878e9f50f58 2021-07-26T06:14:42.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a83a0bc5-6fa4-4572-8fce-7878e9f50f58' 2021-07-26T06:14:42.923 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 a83a0bc5-6fa4-4572-8fce-7878e9f50f58 2021-07-26T06:14:42.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:14:42.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSUv5gnQWBOBAA5UpnVXiQGysrYM1EgJ42FA== 2021-07-26T06:14:42.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSUv5gnQWBOBAA5UpnVXiQGysrYM1EgJ42FA=="}' 2021-07-26T06:14:42.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a83a0bc5-6fa4-4572-8fce-7878e9f50f58 -i td/divergent-priors/0/new.json 2021-07-26T06:14:43.322 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:14:43.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2021-07-26T06:14:43.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDSUv5gnQWBOBAA5UpnVXiQGysrYM1EgJ42FA== --osd-uuid a83a0bc5-6fa4-4572-8fce-7878e9f50f58 2021-07-26T06:14:43.374 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:43.368+0000 7fafae6f5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:43.377 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:43.372+0000 7fafae6f5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:43.378 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:43.372+0000 7fafae6f5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:43.379 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:43.372+0000 7fafae6f5f40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:14:43.379 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:43.372+0000 7fafae6f5f40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:14:43.379 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:43.372+0000 7fafae6f5f40 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-07-26T06:14:43.483 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:43.476+0000 7fafae6f5f40 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-07-26T06:14:49.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-07-26T06:14:49.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:14:49.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:14:49.124 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:14:49.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:14:49.621 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:14:49.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:14:49.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:14:49.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:14:49.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:14:49.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:14:49.658 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:49.656+0000 7f5703371f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:49.660 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:49.660+0000 7f5703371f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:49.664 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:49.660+0000 7f5703371f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:50.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:14:50.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:14:50.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:14:50.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:14:50.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:14:50.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:14:50.027 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:14:50.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:14:50.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:14:50.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:14:50.202 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:50.200+0000 7f5703371f40 -1 Falling back to public interface 2021-07-26T06:14:50.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:14:51.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:14:51.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:14:51.417 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:14:51.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:14:51.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:14:51.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:14:51.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:14:52.456 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:52.455+0000 7f5703371f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:14:52.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:14:52.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:14:52.822 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:14:52.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:14:52.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:14:52.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:14:53.356 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/58847,v1:127.0.0.1:6803/58847] [v2:127.0.0.1:6804/58847,v1:127.0.0.1:6805/58847] exists,up a83a0bc5-6fa4-4572-8fce-7878e9f50f58 2021-07-26T06:14:53.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:14:53.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:14:53.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:14:53.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-07-26T06:14:53.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2021-07-26T06:14:53.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:14:53.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:14:53.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:14:53.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:14:53.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:14:53.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:14:53.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:14:53.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:14:53.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:14:53.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:14:53.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:14:53.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:14:53.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:14:53.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:14:53.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:14:53.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:14:53.366 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:14:53.366 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:14:53.366 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:14:53.366 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:14:53.367 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:14:53.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:53.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:14:53.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:14:53.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:14:53.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:14:53.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:14:53.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:14:53.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:14:53.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:14:53.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:14:53.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:14:53.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:14:53.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:14:53.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:14:53.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ffede5c8-cf1c-400b-9ab1-242a38919ce0 2021-07-26T06:14:53.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ffede5c8-cf1c-400b-9ab1-242a38919ce0' 2021-07-26T06:14:53.372 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 ffede5c8-cf1c-400b-9ab1-242a38919ce0 2021-07-26T06:14:53.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:14:53.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdUv5g8+IAFxAAqmf5I3HH9puRuJTqE1u8/Q== 2021-07-26T06:14:53.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdUv5g8+IAFxAAqmf5I3HH9puRuJTqE1u8/Q=="}' 2021-07-26T06:14:53.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ffede5c8-cf1c-400b-9ab1-242a38919ce0 -i td/divergent-priors/1/new.json 2021-07-26T06:14:53.992 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:14:54.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2021-07-26T06:14:54.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdUv5g8+IAFxAAqmf5I3HH9puRuJTqE1u8/Q== --osd-uuid ffede5c8-cf1c-400b-9ab1-242a38919ce0 2021-07-26T06:14:54.048 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:54.043+0000 7f91db776f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:54.051 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:54.047+0000 7f91db776f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:54.053 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:54.051+0000 7f91db776f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:14:54.053 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:54.051+0000 7f91db776f40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:14:54.054 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:54.051+0000 7f91db776f40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:14:54.054 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:54.051+0000 7f91db776f40 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-07-26T06:14:54.142 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:14:54.139+0000 7f91db776f40 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-07-26T06:15:00.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-07-26T06:15:00.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:15:00.403 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:15:00.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:15:00.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:15:00.923 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:15:00.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:15:00.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:00.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:15:00.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:15:00.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:15:00.961 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:00.959+0000 7f0053f8cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:00.963 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:00.959+0000 7f0053f8cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:00.964 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:00.963+0000 7f0053f8cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:01.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:15:01.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:15:01.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:15:01.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:15:01.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:15:01.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:01.343 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:15:01.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:15:01.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:01.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:01.502 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:01.499+0000 7f0053f8cf40 -1 Falling back to public interface 2021-07-26T06:15:01.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:02.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:02.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:02.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:15:02.735 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:15:02.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:02.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:03.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:04.064 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:04.063+0000 7f0053f8cf40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:15:04.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:04.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:04.146 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:15:04.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:15:04.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:04.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:04.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:05.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:05.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:05.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:15:05.723 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:15:05.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:05.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:15:06.104 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/60227,v1:127.0.0.1:6811/60227] [v2:127.0.0.1:6812/60227,v1:127.0.0.1:6813/60227] exists,up ffede5c8-cf1c-400b-9ab1-242a38919ce0 2021-07-26T06:15:06.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:15:06.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:15:06.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:15:06.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-07-26T06:15:06.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2021-07-26T06:15:06.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2021-07-26T06:15:06.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:15:06.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:15:06.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:15:06.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:15:06.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:15:06.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:15:06.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:15:06.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:15:06.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:15:06.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:15:06.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:15:06.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:15:06.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:15:06.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:15:06.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:15:06.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:15:06.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:15:06.113 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:15:06.113 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:15:06.113 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:15:06.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:15:06.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:15:06.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:15:06.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:15:06.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:15:06.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:15:06.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:15:06.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:15:06.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:15:06.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:06.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:15:06.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:15:06.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:15:06.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:15:06.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93b45a7f-022d-45f0-a0fa-7fb07deeeabd 2021-07-26T06:15:06.119 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 93b45a7f-022d-45f0-a0fa-7fb07deeeabd 2021-07-26T06:15:06.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 93b45a7f-022d-45f0-a0fa-7fb07deeeabd' 2021-07-26T06:15:06.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:15:06.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqUv5gCjM0CBAATbIzJSF5eBG3ku1DuagtCQ== 2021-07-26T06:15:06.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqUv5gCjM0CBAATbIzJSF5eBG3ku1DuagtCQ=="}' 2021-07-26T06:15:06.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93b45a7f-022d-45f0-a0fa-7fb07deeeabd -i td/divergent-priors/2/new.json 2021-07-26T06:15:06.615 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:15:06.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2021-07-26T06:15:06.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDqUv5gCjM0CBAATbIzJSF5eBG3ku1DuagtCQ== --osd-uuid 93b45a7f-022d-45f0-a0fa-7fb07deeeabd 2021-07-26T06:15:06.670 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:06.667+0000 7fbbac97bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:06.683 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:06.679+0000 7fbbac97bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:06.684 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:06.683+0000 7fbbac97bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:06.685 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:06.683+0000 7fbbac97bf40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:15:06.685 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:06.683+0000 7fbbac97bf40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:15:06.686 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:06.683+0000 7fbbac97bf40 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-07-26T06:15:06.729 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:06.727+0000 7fbbac97bf40 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-07-26T06:15:12.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-07-26T06:15:12.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:15:12.879 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:15:12.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:15:12.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:15:13.356 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:15:13.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:15:13.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:15:13.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:15:13.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:15:13.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:15:13.391 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:13.386+0000 7f032561bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:13.393 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:13.390+0000 7f032561bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:13.396 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:13.390+0000 7f032561bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:15:13.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:15:13.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:15:13.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:15:13.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:15:13.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:15:13.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:13.730 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:15:13.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:15:13.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:13.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:13.947 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:13.942+0000 7f032561bf40 -1 Falling back to public interface 2021-07-26T06:15:14.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:15.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:15.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:15.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:15:15.125 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:15:15.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:15.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:15.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:16.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:16.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:16.487 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:15:16.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:15:16.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:16.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:16.497 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:15:16.494+0000 7f032561bf40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:15:16.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:15:17.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:15:17.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:15:17.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:15:17.865 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:15:17.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:15:17.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:15:18.232 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/61636,v1:127.0.0.1:6819/61636] [v2:127.0.0.1:6820/61636,v1:127.0.0.1:6821/61636] exists,up 93b45a7f-022d-45f0-a0fa-7fb07deeeabd 2021-07-26T06:15:18.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:15:18.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:15:18.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:15:18.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2021-07-26T06:15:19.082 INFO:tasks.workunit.client.0.smithi094.stderr:noout is set 2021-07-26T06:15:19.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2021-07-26T06:15:20.022 INFO:tasks.workunit.client.0.smithi094.stderr:noin is set 2021-07-26T06:15:20.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2021-07-26T06:15:20.862 INFO:tasks.workunit.client.0.smithi094.stderr:nodown is set 2021-07-26T06:15:20.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2021-07-26T06:15:20.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: local pool_name=test 2021-07-26T06:15:20.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: shift 2021-07-26T06:15:20.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: local allow_overwrites=true 2021-07-26T06:15:20.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: shift 2021-07-26T06:15:20.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-07-26T06:15:21.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: create_ec_pool: create_pool test 1 1 erasure myprofile 2021-07-26T06:15:21.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-07-26T06:15:22.258 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T06:15:22.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:15:23.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: '[' true = true ']' 2021-07-26T06:15:23.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2021-07-26T06:15:24.319 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 allow_ec_overwrites to true 2021-07-26T06:15:24.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: create_ec_pool: wait_for_clean 2021-07-26T06:15:24.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:15:24.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:15:24.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:15:24.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:15:24.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:15:24.334 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:15:24.334 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:15:24.335 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:15:24.335 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:15:24.335 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:15:24.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:15:24.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:15:24.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:15:24.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:15:24.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:15:25.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:15:25.001 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:15:25.001 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:15:25.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:15:25.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:25.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:15:25.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836489 2021-07-26T06:15:25.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836489 2021-07-26T06:15:25.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489' 2021-07-26T06:15:25.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:25.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:15:25.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:15:25.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:15:25.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2021-07-26T06:15:25.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:25.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:15:25.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509444 2021-07-26T06:15:25.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509444 2021-07-26T06:15:25.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2021-07-26T06:15:25.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:25.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836489 2021-07-26T06:15:25.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:25.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:15:25.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836489 2021-07-26T06:15:25.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:25.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836489 2021-07-26T06:15:25.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-07-26T06:15:25.474 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836489 2021-07-26T06:15:25.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:15:25.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836489 2021-07-26T06:15:25.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:15:26.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:15:26.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:15:27.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836489 2021-07-26T06:15:27.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:27.290 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:15:27.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:27.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:15:27.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:15:27.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:27.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:15:27.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:15:27.297 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672966 2021-07-26T06:15:27.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:15:27.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672967 -lt 42949672966 2021-07-26T06:15:27.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:27.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509444 2021-07-26T06:15:27.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:27.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:15:27.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509444 2021-07-26T06:15:27.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:27.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509444 2021-07-26T06:15:27.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-07-26T06:15:27.721 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509444 2021-07-26T06:15:27.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:15:28.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509444 -lt 64424509444 2021-07-26T06:15:28.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:15:28.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:15:28.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:15:28.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:15:28.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:15:28.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:15:28.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:15:28.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:15:28.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:15:28.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:15:28.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:15:29.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:15:29.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:15:29.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:15:29.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:15:29.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:15:29.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:15:29.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:15:29.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: create_ec_pool: return 0 2021-07-26T06:15:29.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2021-07-26T06:15:29.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:15:29.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:15:29.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:15:29.942 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:15:29.942 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:15:29.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:15:29.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:29.943 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:15:30.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836492 2021-07-26T06:15:30.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836492 2021-07-26T06:15:30.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492' 2021-07-26T06:15:30.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:30.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:15:30.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672969 2021-07-26T06:15:30.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672969 2021-07-26T06:15:30.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2021-07-26T06:15:30.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:30.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:15:30.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509447 2021-07-26T06:15:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509447 2021-07-26T06:15:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492 1-42949672969 2-64424509447' 2021-07-26T06:15:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:30.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836492 2021-07-26T06:15:30.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:30.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:15:30.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836492 2021-07-26T06:15:30.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:30.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836492 2021-07-26T06:15:30.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-07-26T06:15:30.399 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836492 2021-07-26T06:15:30.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:15:30.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836492 2021-07-26T06:15:30.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:15:31.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:15:31.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:15:32.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836492 -lt 21474836492 2021-07-26T06:15:32.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:32.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672969 2021-07-26T06:15:32.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:32.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:15:32.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672969 2021-07-26T06:15:32.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:32.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672969 2021-07-26T06:15:32.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-07-26T06:15:32.159 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672969 2021-07-26T06:15:32.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:15:32.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672969 2021-07-26T06:15:32.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:32.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509447 2021-07-26T06:15:32.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:32.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:15:32.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509447 2021-07-26T06:15:32.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:32.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509447 2021-07-26T06:15:32.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-07-26T06:15:32.557 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509447 2021-07-26T06:15:32.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:15:32.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509447 -lt 64424509447 2021-07-26T06:15:32.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2021-07-26T06:15:32.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:15:32.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:15:32.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:15:32.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:15:32.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:15:32.963 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:15:32.964 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:15:32.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:15:32.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:15:32.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:15:33.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:15:33.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:15:33.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:15:33.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:15:33.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:15:33.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:15:33.554 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:15:33.555 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:15:33.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:15:33.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:33.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:15:33.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836494 2021-07-26T06:15:33.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836494 2021-07-26T06:15:33.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494' 2021-07-26T06:15:33.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:33.742 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:15:33.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672972 2021-07-26T06:15:33.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672972 2021-07-26T06:15:33.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672972' 2021-07-26T06:15:33.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:15:33.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:15:34.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509450 2021-07-26T06:15:34.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509450 2021-07-26T06:15:34.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836494 1-42949672972 2-64424509450' 2021-07-26T06:15:34.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:34.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836494 2021-07-26T06:15:34.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:34.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:15:34.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836494 2021-07-26T06:15:34.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:34.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836494 2021-07-26T06:15:34.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2021-07-26T06:15:34.073 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836494 2021-07-26T06:15:34.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:15:34.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836492 -lt 21474836494 2021-07-26T06:15:34.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:15:35.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:15:35.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:15:35.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836494 2021-07-26T06:15:35.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:35.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672972 2021-07-26T06:15:35.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:35.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:15:35.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:35.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672972 2021-07-26T06:15:35.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672972 2021-07-26T06:15:35.822 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672972 2021-07-26T06:15:35.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2021-07-26T06:15:35.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:15:36.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672972 -lt 42949672972 2021-07-26T06:15:36.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:15:36.242 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509450 2021-07-26T06:15:36.242 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:15:36.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:15:36.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509450 2021-07-26T06:15:36.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:15:36.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509450 2021-07-26T06:15:36.248 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509450 2021-07-26T06:15:36.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2021-07-26T06:15:36.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:15:36.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509450 -lt 64424509450 2021-07-26T06:15:36.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:15:36.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:15:36.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:15:37.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:15:37.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:15:37.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:15:37.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:15:37.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:15:37.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:15:37.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:15:37.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:15:37.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:15:37.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:15:37.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:15:37.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:15:37.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:15:37.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:15:37.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:15:37.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2021-07-26T06:15:37.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2021-07-26T06:15:38.348 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:15:38.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2021-07-26T06:15:38.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2021-07-26T06:15:38.357 INFO:tasks.workunit.client.0.smithi094.stdout:primary and soon to be divergent is 1 2021-07-26T06:15:38.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:15:38.671 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:15:38.672 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:15:23.581984+0000 0'0 26:10 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:15:38.672 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:15:38.672 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:15:38.673 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:15:38.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2021-07-26T06:15:38.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-07-26T06:15:38.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2021-07-26T06:15:38.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2021-07-26T06:15:38.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-07-26T06:15:38.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2021-07-26T06:15:38.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2021-07-26T06:15:38.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-07-26T06:15:38.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2021-07-26T06:15:38.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2021-07-26T06:15:38.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2021-07-26T06:15:38.685 INFO:tasks.workunit.client.0.smithi094.stdout:writing initial objects 2021-07-26T06:15:38.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2021-07-26T06:15:38.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:38.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2021-07-26T06:15:38.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:38.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2021-07-26T06:15:39.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:39.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2021-07-26T06:15:39.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:39.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2021-07-26T06:15:39.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:39.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2021-07-26T06:15:39.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:39.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2021-07-26T06:15:39.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:39.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2021-07-26T06:15:40.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:40.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2021-07-26T06:15:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:40.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2021-07-26T06:15:40.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:40.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2021-07-26T06:15:40.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:40.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2021-07-26T06:15:41.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:41.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2021-07-26T06:15:41.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:41.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2021-07-26T06:15:41.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:41.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2021-07-26T06:15:41.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:41.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2021-07-26T06:15:42.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:42.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2021-07-26T06:15:42.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:42.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2021-07-26T06:15:42.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:42.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2021-07-26T06:15:42.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:42.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2021-07-26T06:15:43.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:43.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2021-07-26T06:15:43.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:43.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2021-07-26T06:15:43.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:43.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2021-07-26T06:15:43.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:43.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2021-07-26T06:15:44.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:44.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2021-07-26T06:15:44.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:44.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2021-07-26T06:15:44.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:44.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2021-07-26T06:15:44.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:44.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2021-07-26T06:15:44.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:44.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2021-07-26T06:15:45.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:45.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2021-07-26T06:15:45.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:45.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2021-07-26T06:15:45.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:45.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2021-07-26T06:15:45.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:45.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2021-07-26T06:15:45.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:45.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2021-07-26T06:15:46.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:46.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2021-07-26T06:15:46.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:46.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2021-07-26T06:15:46.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:46.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2021-07-26T06:15:47.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:47.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2021-07-26T06:15:47.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:47.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2021-07-26T06:15:47.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:47.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2021-07-26T06:15:47.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:47.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2021-07-26T06:15:47.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:47.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2021-07-26T06:15:48.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:48.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2021-07-26T06:15:48.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:48.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2021-07-26T06:15:48.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:48.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2021-07-26T06:15:48.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:48.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2021-07-26T06:15:49.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:49.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2021-07-26T06:15:49.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:49.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2021-07-26T06:15:49.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:49.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2021-07-26T06:15:49.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:49.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2021-07-26T06:15:50.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:50.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2021-07-26T06:15:50.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:50.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2021-07-26T06:15:50.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:50.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2021-07-26T06:15:50.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:50.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2021-07-26T06:15:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:50.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2021-07-26T06:15:51.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:51.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2021-07-26T06:15:51.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:51.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2021-07-26T06:15:51.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:51.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2021-07-26T06:15:51.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:51.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2021-07-26T06:15:52.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:52.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2021-07-26T06:15:52.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:52.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2021-07-26T06:15:52.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:52.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2021-07-26T06:15:52.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:52.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2021-07-26T06:15:52.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:52.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2021-07-26T06:15:53.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:53.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2021-07-26T06:15:53.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:53.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2021-07-26T06:15:53.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:53.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2021-07-26T06:15:53.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:53.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2021-07-26T06:15:54.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:54.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2021-07-26T06:15:54.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:54.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2021-07-26T06:15:54.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:54.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2021-07-26T06:15:54.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:54.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2021-07-26T06:15:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2021-07-26T06:15:55.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:55.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2021-07-26T06:15:55.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:55.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2021-07-26T06:15:55.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:55.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2021-07-26T06:15:55.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:55.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2021-07-26T06:15:56.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:56.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2021-07-26T06:15:56.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:56.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2021-07-26T06:15:56.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:56.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2021-07-26T06:15:56.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:56.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2021-07-26T06:15:57.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:57.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2021-07-26T06:15:57.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:57.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2021-07-26T06:15:57.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:57.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2021-07-26T06:15:57.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:57.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2021-07-26T06:15:58.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:58.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2021-07-26T06:15:58.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:58.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2021-07-26T06:15:58.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:58.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2021-07-26T06:15:58.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:58.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2021-07-26T06:15:58.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:58.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2021-07-26T06:15:59.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:59.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2021-07-26T06:15:59.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:59.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2021-07-26T06:15:59.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:15:59.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2021-07-26T06:16:00.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:00.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2021-07-26T06:16:00.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:00.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2021-07-26T06:16:00.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:00.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2021-07-26T06:16:00.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:00.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2021-07-26T06:16:00.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:00.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2021-07-26T06:16:01.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:01.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2021-07-26T06:16:01.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:01.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-07-26T06:16:01.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-07-26T06:16:01.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-07-26T06:16:02.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:16:02.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2021-07-26T06:16:02.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:16:02.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:16:02.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:16:02.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:16:02.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:16:02.022 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:16:02.022 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:16:02.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:16:02.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:16:02.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:16:02.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:16:02.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:16:02.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:16:02.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:16:02.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:16:02.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:16:02.529 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:16:02.530 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:16:02.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:16:02.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:02.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:16:02.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836502 2021-07-26T06:16:02.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836502 2021-07-26T06:16:02.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836502' 2021-07-26T06:16:02.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:02.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:16:02.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672980 2021-07-26T06:16:02.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672980 2021-07-26T06:16:02.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836502 1-42949672980' 2021-07-26T06:16:02.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:02.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:16:03.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509458 2021-07-26T06:16:03.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509458 2021-07-26T06:16:03.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509458' 2021-07-26T06:16:03.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:03.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836502 2021-07-26T06:16:03.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:03.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:16:03.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836502 2021-07-26T06:16:03.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:03.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836502 2021-07-26T06:16:03.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2021-07-26T06:16:03.049 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836502 2021-07-26T06:16:03.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:16:03.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836502 -lt 21474836502 2021-07-26T06:16:03.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:03.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672980 2021-07-26T06:16:03.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:03.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:16:03.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672980 2021-07-26T06:16:03.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:03.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672980 2021-07-26T06:16:03.418 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672980 2021-07-26T06:16:03.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-07-26T06:16:03.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:16:03.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672978 -lt 42949672980 2021-07-26T06:16:03.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:16:04.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:16:04.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:16:05.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672980 -lt 42949672980 2021-07-26T06:16:05.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:05.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509458 2021-07-26T06:16:05.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:05.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:16:05.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509458 2021-07-26T06:16:05.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:05.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509458 2021-07-26T06:16:05.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2021-07-26T06:16:05.209 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509458 2021-07-26T06:16:05.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:16:05.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509458 -lt 64424509458 2021-07-26T06:16:05.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:16:05.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:05.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:06.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:16:06.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:06.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:06.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:06.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:06.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:06.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:06.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:06.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:16:06.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:06.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:06.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:06.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:16:06.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:16:06.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:16:06.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2021-07-26T06:16:06.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T06:16:06.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=existing_1 2021-07-26T06:16:06.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test existing_1 2021-07-26T06:16:06.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T06:16:07.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2021-07-26T06:16:07.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2021-07-26T06:16:07.340 INFO:tasks.workunit.client.0.smithi094.stdout:blackholing osds 0 2 2021-07-26T06:16:07.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:16:07.685 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:16:07.686 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2021-07-26T06:15:23.581984+0000 29'100 29:110 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:16:07.686 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:16:07.687 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:16:07.688 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:16:07.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:07.699 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2021-07-26T06:16:07.699 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:16:07.699 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:16:07.700 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:16:07.700 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:16:07.700 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:16:07.700 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:16:07.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-07-26T06:16:07.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok config set objectstore_blackhole 1 2021-07-26T06:16:07.775 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:16:07.775 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:16:07.775 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:16:07.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:07.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2021-07-26T06:16:07.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:16:07.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:16:07.784 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:16:07.785 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:16:07.785 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:16:07.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:16:07.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-07-26T06:16:07.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok config set objectstore_blackhole 1 2021-07-26T06:16:07.863 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:16:07.864 INFO:tasks.workunit.client.0.smithi094.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-07-26T06:16:07.864 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:16:07.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2021-07-26T06:16:07.870 INFO:tasks.workunit.client.0.smithi094.stdout:writing divergent object 2021-07-26T06:16:07.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2021-07-26T06:16:07.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2021-07-26T06:16:08.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2021-07-26T06:16:08.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-07-26T06:16:09.151 INFO:tasks.workunit.client.0.smithi094.stdout:created pool test snap snap1 2021-07-26T06:16:09.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2021-07-26T06:16:09.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2021-07-26T06:16:09.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-07-26T06:16:19.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2021-07-26T06:16:19.168 INFO:tasks.workunit.client.0.smithi094.stderr:rados: no process found 2021-07-26T06:16:19.169 INFO:tasks.workunit.client.0.smithi094.stdout:killing all the osds 2021-07-26T06:16:19.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2021-07-26T06:16:19.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:16:19.510 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:16:19.510 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 0 0 0 62317 0 0 14 14 active+clean 2021-07-26T06:15:23.581984+0000 30'104 30:114 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:16:19.511 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:16:19.511 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:16:19.513 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:16:19.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2021-07-26T06:16:19.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:16:19.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:16:19.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:16:19.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:16:19.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:16:19.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:16:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-07-26T06:16:19.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2021-07-26T06:16:20.190 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:16:20.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-07-26T06:16:20.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2021-07-26T06:16:20.751 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T06:16:20.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-07-26T06:16:20.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2021-07-26T06:16:21.363 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already down. 2021-07-26T06:16:21.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:21.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2021-07-26T06:16:21.993 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T06:16:22.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:22.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2021-07-26T06:16:22.621 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already out. 2021-07-26T06:16:22.629 INFO:tasks.workunit.client.0.smithi094.stdout:bringing up non_divergent 0 2 2021-07-26T06:16:22.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2021-07-26T06:16:22.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:16:23.009 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:16:23.009 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 0 0 0 62317 0 0 14 14 stale+active+clean 2021-07-26T06:15:23.581984+0000 30'104 30:114 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:16:23.010 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:16:23.010 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:16:23.011 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:16:23.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:23.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-07-26T06:16:23.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:16:23.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:16:23.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:16:23.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:16:23.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:16:23.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:16:23.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:16:23.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:16:23.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:16:23.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:16:23.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:16:23.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:16:23.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:16:23.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:16:23.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:16:23.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:16:23.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:16:23.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:16:23.028 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:16:23.029 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:16:23.029 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:16:23.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:16:23.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:16:23.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:16:23.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:16:23.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:16:23.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:16:23.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:16:23.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:16:23.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:16:23.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:16:23.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:16:23.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:16:23.033 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:16:23.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:16:23.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/0/whoami 2021-07-26T06:16:23.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:16:23.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:16:23.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:16:23.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:16:23.061 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:23.055+0000 7f5b92a99f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:23.064 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:23.059+0000 7f5b92a99f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:23.067 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:23.059+0000 7f5b92a99f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:23.357 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:23.351+0000 7f5b92a99f40 -1 Falling back to public interface 2021-07-26T06:16:23.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:16:23.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:16:23.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:16:23.416 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:16:23.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:16:23.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:16:23.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:23.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:16:23.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:16:23.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:23.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:24.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:24.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:24.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:16:24.816 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:16:24.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:24.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:16:25.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:26.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:26.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:26.179 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:16:26.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:16:26.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:26.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:16:26.275 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:26.263+0000 7f5b92a99f40 -1 osd.0 29 log_to_monitors {default=true} 2021-07-26T06:16:26.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:27.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:27.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:27.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:16:27.560 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:16:27.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:27.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:16:27.960 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/66091,v1:127.0.0.1:6803/66091] [v2:127.0.0.1:6804/66091,v1:127.0.0.1:6805/66091] exists,up a83a0bc5-6fa4-4572-8fce-7878e9f50f58 2021-07-26T06:16:27.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:16:27.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:16:27.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:16:27.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:27.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-07-26T06:16:27.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:16:27.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:16:27.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:16:27.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:16:27.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:16:27.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:16:27.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:16:27.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:16:27.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:16:27.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:16:27.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:16:27.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:16:27.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:16:27.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:16:27.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:16:27.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:16:27.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:16:27.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:16:27.971 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:16:27.972 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:16:27.972 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:16:27.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:16:27.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:16:27.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:16:27.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:16:27.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:16:27.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:16:27.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:16:27.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:16:27.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:16:27.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:16:27.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:16:27.975 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:16:27.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:16:27.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:16:27.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/2/whoami 2021-07-26T06:16:27.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:16:27.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:16:27.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:16:27.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:16:28.003 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:27.995+0000 7f7061f2cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:28.014 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:28.007+0000 7f7061f2cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:28.016 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:28.011+0000 7f7061f2cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:16:28.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:16:28.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:16:28.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:16:28.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:16:28.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:16:28.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:28.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:16:28.368 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:16:28.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:28.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:16:28.549 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:28.542+0000 7f7061f2cf40 -1 Falling back to public interface 2021-07-26T06:16:28.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:29.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:29.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:29.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:16:29.748 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:16:29.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:29.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:16:30.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:31.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:31.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:31.132 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:16:31.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:16:31.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:31.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:16:31.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:16:31.632 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:16:31.626+0000 7f7061f2cf40 -1 osd.2 29 log_to_monitors {default=true} 2021-07-26T06:16:32.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:16:32.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:16:32.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:16:32.518 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:16:32.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:16:32.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:16:32.886 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/66669,v1:127.0.0.1:6811/66669] [v2:127.0.0.1:6812/66669,v1:127.0.0.1:6813/66669] exists,up 93b45a7f-022d-45f0-a0fa-7fb07deeeabd 2021-07-26T06:16:32.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:16:32.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:16:32.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:16:32.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:32.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2021-07-26T06:16:33.489 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already in. 2021-07-26T06:16:33.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:16:33.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2021-07-26T06:16:34.212 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already in. 2021-07-26T06:16:34.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2021-07-26T06:16:39.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2021-07-26T06:16:39.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2021-07-26T06:16:39.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2021-07-26T06:16:39.230 INFO:tasks.workunit.client.0.smithi094.stdout:writing non-divergent object existing_4 2021-07-26T06:16:39.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:16:39.579 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:16:39.579 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2021-07-26T06:16:34.708444+0000 29'100 42:119 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:16:39.580 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:16:39.580 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:16:39.582 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:16:39.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2021-07-26T06:16:39.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:16:39.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2021-07-26T06:16:39.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:16:39.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:16:39.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:16:39.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:16:39.743 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:16:39.743 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:16:39.744 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:16:39.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:16:39.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:16:39.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:16:39.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:16:39.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:16:39.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:16:39.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:16:39.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:16:40.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:16:40.283 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:16:40.283 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:16:40.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:16:40.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:40.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:16:40.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=158913789957 2021-07-26T06:16:40.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 158913789957 2021-07-26T06:16:40.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-158913789957' 2021-07-26T06:16:40.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:40.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:16:40.585 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-07-26T06:16:40.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T06:16:40.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T06:16:40.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T06:16:40.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:16:40.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:16:40.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=167503724548 2021-07-26T06:16:40.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 167503724548 2021-07-26T06:16:40.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2021-07-26T06:16:40.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:40.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-158913789957 2021-07-26T06:16:40.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:40.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:16:40.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-158913789957 2021-07-26T06:16:40.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:40.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=158913789957 2021-07-26T06:16:40.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2021-07-26T06:16:40.744 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 158913789957 2021-07-26T06:16:40.745 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:16:41.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 158913789957 -lt 158913789957 2021-07-26T06:16:41.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:16:41.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-167503724548 2021-07-26T06:16:41.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:16:41.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:16:41.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-167503724548 2021-07-26T06:16:41.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:16:41.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=167503724548 2021-07-26T06:16:41.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2021-07-26T06:16:41.125 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 167503724548 2021-07-26T06:16:41.126 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:16:41.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 167503724548 -lt 167503724548 2021-07-26T06:16:41.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:16:41.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:41.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:41.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:16:41.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:41.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:41.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:41.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:41.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:41.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:41.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:42.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:16:42.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:42.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:42.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:16:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:16:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:16:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:16:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:16:42.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:16:42.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:16:42.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:42.894 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:42.895 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:42.895 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:42.895 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:42.895 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:42.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:43.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:16:43.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:43.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:43.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:43.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:16:43.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:16:43.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:16:43.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:16:43.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:16:43.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:16:43.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:16:43.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:16:43.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:16:44.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:16:44.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:16:44.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:16:44.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:16:44.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:16:44.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:16:44.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:44.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:44.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:44.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:44.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:44.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:44.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:44.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:16:44.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:44.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:44.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:45.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:16:45.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:16:45.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:16:45.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:16:45.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:16:45.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:16:45.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:16:45.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:16:45.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:16:45.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:16:45.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:16:45.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:16:45.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:16:45.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:16:46.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:16:46.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:46.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:46.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:46.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:46.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:46.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:46.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:46.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:16:46.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:46.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:46.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:47.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:16:47.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:16:47.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:16:47.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:16:47.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:16:47.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:16:47.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:16:47.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:16:47.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:16:47.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:16:47.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:16:47.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:16:47.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:16:47.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:16:48.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:16:48.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:48.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:48.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:48.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:48.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:48.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:48.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:48.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:16:48.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:48.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:48.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:49.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:16:49.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:16:49.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:16:49.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:16:49.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:16:49.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:16:49.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:16:49.189 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:16:49.189 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:16:49.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:16:49.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:16:49.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 4 >= 8 )) 2021-07-26T06:16:49.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:16:49.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 1.6 2021-07-26T06:16:51.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:16:51.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:51.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:51.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:51.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:51.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:51.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:51.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:51.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:16:51.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:51.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:51.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:52.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:16:52.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:16:52.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:16:52.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:16:52.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:16:52.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:16:52.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:16:52.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:16:52.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:16:52.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:16:52.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:16:52.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 5 >= 8 )) 2021-07-26T06:16:52.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:16:52.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 3.2 2021-07-26T06:16:55.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:16:55.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:16:55.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:16:55.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:16:55.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:16:55.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:16:55.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:16:55.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:16:56.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:16:56.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:16:56.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:16:56.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:16:56.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:16:56.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:16:56.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:16:56.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:16:56.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:16:56.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:16:56.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:16:56.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:16:56.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:16:57.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:16:57.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:16:57.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 6 >= 8 )) 2021-07-26T06:16:57.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:16:57.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 6.4 2021-07-26T06:17:03.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:03.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:03.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:03.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:03.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:03.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:03.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:03.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:03.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:03.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:03.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:03.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:04.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:04.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:04.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:04.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:04.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:04.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:04.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:04.345 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:04.345 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:04.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:04.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:04.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 7 >= 8 )) 2021-07-26T06:17:04.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:04.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 7.3 2021-07-26T06:17:12.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:12.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:12.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:12.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:12.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:12.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:12.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:12.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:12.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:12.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:12.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:12.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:13.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:13.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:13.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:13.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:13.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:13.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:13.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:13.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:13.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 8 >= 8 )) 2021-07-26T06:17:13.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph report 2021-07-26T06:17:14.008 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:17:14.008 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_fingerprint": "4722d731-e412-4982-951f-8bd4beca4e17", 2021-07-26T06:17:14.008 INFO:tasks.workunit.client.0.smithi094.stdout: "version": "17.0.0-6358-g197e564f", 2021-07-26T06:17:14.009 INFO:tasks.workunit.client.0.smithi094.stdout: "commit": "197e564fdaee12596de1dc8f3253d1704ae66d5d", 2021-07-26T06:17:14.009 INFO:tasks.workunit.client.0.smithi094.stdout: "timestamp": "2021-07-26T06:17:13.998736+0000", 2021-07-26T06:17:14.009 INFO:tasks.workunit.client.0.smithi094.stdout: "tag": "", 2021-07-26T06:17:14.009 INFO:tasks.workunit.client.0.smithi094.stdout: "health": { 2021-07-26T06:17:14.010 INFO:tasks.workunit.client.0.smithi094.stdout: "status": "HEALTH_WARN", 2021-07-26T06:17:14.010 INFO:tasks.workunit.client.0.smithi094.stdout: "checks": { 2021-07-26T06:17:14.010 INFO:tasks.workunit.client.0.smithi094.stdout: "OSDMAP_FLAGS": { 2021-07-26T06:17:14.010 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:14.011 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:14.011 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "nodown,noout,noin flag(s) set", 2021-07-26T06:17:14.011 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 17 2021-07-26T06:17:14.011 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.011 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [], 2021-07-26T06:17:14.012 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:14.012 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.012 INFO:tasks.workunit.client.0.smithi094.stdout: "OSD_DOWN": { 2021-07-26T06:17:14.012 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:14.013 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:14.013 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 osds down", 2021-07-26T06:17:14.013 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:17:14.013 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.014 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:17:14.014 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.014 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "osd.1 (root=default,host=smithi094) is down" 2021-07-26T06:17:14.014 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.014 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.015 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:14.015 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.015 INFO:tasks.workunit.client.0.smithi094.stdout: "PG_DEGRADED": { 2021-07-26T06:17:14.015 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:14.016 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:14.016 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-07-26T06:17:14.016 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:17:14.017 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.017 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:17:14.017 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.017 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2021-07-26T06:17:14.018 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.018 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.018 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:14.018 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.018 INFO:tasks.workunit.client.0.smithi094.stdout: "POOL_APP_NOT_ENABLED": { 2021-07-26T06:17:14.019 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:14.019 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:14.019 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 pool(s) do not have an application enabled", 2021-07-26T06:17:14.019 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:17:14.020 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.020 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:17:14.020 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.020 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "application not enabled on pool 'test'" 2021-07-26T06:17:14.020 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.021 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.021 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-07-26T06:17:14.021 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.021 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.022 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:14.022 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.022 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.022 INFO:tasks.workunit.client.0.smithi094.stdout: "mutes": [] 2021-07-26T06:17:14.023 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.023 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_first_committed": 1, 2021-07-26T06:17:14.023 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_last_committed": 1, 2021-07-26T06:17:14.023 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap": { 2021-07-26T06:17:14.023 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:17:14.024 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:17:14.024 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:14:41.137869Z", 2021-07-26T06:17:14.024 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:14:41.137869Z", 2021-07-26T06:17:14.024 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release": 17, 2021-07-26T06:17:14.025 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release_name": "quincy", 2021-07-26T06:17:14.025 INFO:tasks.workunit.client.0.smithi094.stdout: "election_strategy": 1, 2021-07-26T06:17:14.025 INFO:tasks.workunit.client.0.smithi094.stdout: "disallowed_leaders: ": "", 2021-07-26T06:17:14.025 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": false, 2021-07-26T06:17:14.025 INFO:tasks.workunit.client.0.smithi094.stdout: "features": { 2021-07-26T06:17:14.026 INFO:tasks.workunit.client.0.smithi094.stdout: "persistent": [ 2021-07-26T06:17:14.026 INFO:tasks.workunit.client.0.smithi094.stdout: "kraken", 2021-07-26T06:17:14.027 INFO:tasks.workunit.client.0.smithi094.stdout: "luminous", 2021-07-26T06:17:14.027 INFO:tasks.workunit.client.0.smithi094.stdout: "mimic", 2021-07-26T06:17:14.027 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap-prune", 2021-07-26T06:17:14.027 INFO:tasks.workunit.client.0.smithi094.stdout: "nautilus", 2021-07-26T06:17:14.027 INFO:tasks.workunit.client.0.smithi094.stdout: "octopus", 2021-07-26T06:17:14.028 INFO:tasks.workunit.client.0.smithi094.stdout: "pacific", 2021-07-26T06:17:14.028 INFO:tasks.workunit.client.0.smithi094.stdout: "elector-pinging", 2021-07-26T06:17:14.028 INFO:tasks.workunit.client.0.smithi094.stdout: "quincy" 2021-07-26T06:17:14.028 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.029 INFO:tasks.workunit.client.0.smithi094.stdout: "optional": [] 2021-07-26T06:17:14.029 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.029 INFO:tasks.workunit.client.0.smithi094.stdout: "mons": [ 2021-07-26T06:17:14.029 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.030 INFO:tasks.workunit.client.0.smithi094.stdout: "rank": 0, 2021-07-26T06:17:14.030 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "a", 2021-07-26T06:17:14.030 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:14.030 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.031 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.031 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.031 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115", 2021-07-26T06:17:14.032 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 0 2021-07-26T06:17:14.032 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.032 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.032 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.032 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115/0", 2021-07-26T06:17:14.033 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:7115/0", 2021-07-26T06:17:14.033 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 0, 2021-07-26T06:17:14.033 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 0, 2021-07-26T06:17:14.033 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_location": "{}" 2021-07-26T06:17:14.034 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.034 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.034 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.034 INFO:tasks.workunit.client.0.smithi094.stdout: "quorum": [ 2021-07-26T06:17:14.034 INFO:tasks.workunit.client.0.smithi094.stdout: 0 2021-07-26T06:17:14.035 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.035 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap": { 2021-07-26T06:17:14.035 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 42, 2021-07-26T06:17:14.035 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:17:14.036 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:14:42.057212+0000", 2021-07-26T06:17:14.036 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:16:34.288955+0000", 2021-07-26T06:17:14.036 INFO:tasks.workunit.client.0.smithi094.stdout: "last_up_change": "2021-07-26T06:16:31.821154+0000", 2021-07-26T06:17:14.036 INFO:tasks.workunit.client.0.smithi094.stdout: "last_in_change": "2021-07-26T06:16:33.869345+0000", 2021-07-26T06:17:14.037 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-07-26T06:17:14.037 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_num": 5800384, 2021-07-26T06:17:14.037 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_set": [ 2021-07-26T06:17:14.037 INFO:tasks.workunit.client.0.smithi094.stdout: "nodown", 2021-07-26T06:17:14.038 INFO:tasks.workunit.client.0.smithi094.stdout: "noin", 2021-07-26T06:17:14.038 INFO:tasks.workunit.client.0.smithi094.stdout: "noout", 2021-07-26T06:17:14.038 INFO:tasks.workunit.client.0.smithi094.stdout: "pglog_hardlimit", 2021-07-26T06:17:14.038 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snapdirs", 2021-07-26T06:17:14.039 INFO:tasks.workunit.client.0.smithi094.stdout: "recovery_deletes", 2021-07-26T06:17:14.040 INFO:tasks.workunit.client.0.smithi094.stdout: "sortbitwise" 2021-07-26T06:17:14.040 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.040 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_version": 10, 2021-07-26T06:17:14.041 INFO:tasks.workunit.client.0.smithi094.stdout: "full_ratio": 0.99000000953674316, 2021-07-26T06:17:14.041 INFO:tasks.workunit.client.0.smithi094.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-07-26T06:17:14.041 INFO:tasks.workunit.client.0.smithi094.stdout: "nearfull_ratio": 0.85000002384185791, 2021-07-26T06:17:14.041 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_snapshot": "", 2021-07-26T06:17:14.042 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_max": 1, 2021-07-26T06:17:14.042 INFO:tasks.workunit.client.0.smithi094.stdout: "max_osd": 3, 2021-07-26T06:17:14.042 INFO:tasks.workunit.client.0.smithi094.stdout: "require_min_compat_client": "luminous", 2021-07-26T06:17:14.042 INFO:tasks.workunit.client.0.smithi094.stdout: "min_compat_client": "jewel", 2021-07-26T06:17:14.042 INFO:tasks.workunit.client.0.smithi094.stdout: "require_osd_release": "quincy", 2021-07-26T06:17:14.043 INFO:tasks.workunit.client.0.smithi094.stdout: "pools": [ 2021-07-26T06:17:14.043 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.043 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:17:14.044 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_name": "test", 2021-07-26T06:17:14.045 INFO:tasks.workunit.client.0.smithi094.stdout: "create_time": "2021-07-26T06:15:21.917164+0000", 2021-07-26T06:17:14.045 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": 16389, 2021-07-26T06:17:14.045 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-07-26T06:17:14.046 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 3, 2021-07-26T06:17:14.046 INFO:tasks.workunit.client.0.smithi094.stdout: "size": 3, 2021-07-26T06:17:14.046 INFO:tasks.workunit.client.0.smithi094.stdout: "min_size": 2, 2021-07-26T06:17:14.047 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_rule": 1, 2021-07-26T06:17:14.047 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_count": 0, 2021-07-26T06:17:14.048 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_target": 0, 2021-07-26T06:17:14.048 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_barrier": 0, 2021-07-26T06:17:14.048 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-07-26T06:17:14.048 INFO:tasks.workunit.client.0.smithi094.stdout: "object_hash": 2, 2021-07-26T06:17:14.049 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_autoscale_mode": "off", 2021-07-26T06:17:14.049 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num": 1, 2021-07-26T06:17:14.049 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num": 1, 2021-07-26T06:17:14.049 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num_target": 1, 2021-07-26T06:17:14.049 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_target": 1, 2021-07-26T06:17:14.050 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_pending": 1, 2021-07-26T06:17:14.050 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pg_merge_meta": { 2021-07-26T06:17:14.050 INFO:tasks.workunit.client.0.smithi094.stdout: "source_pgid": "0.0", 2021-07-26T06:17:14.050 INFO:tasks.workunit.client.0.smithi094.stdout: "ready_epoch": 0, 2021-07-26T06:17:14.051 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_started": 0, 2021-07-26T06:17:14.051 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": 0, 2021-07-26T06:17:14.051 INFO:tasks.workunit.client.0.smithi094.stdout: "source_version": "0'0", 2021-07-26T06:17:14.051 INFO:tasks.workunit.client.0.smithi094.stdout: "target_version": "0'0" 2021-07-26T06:17:14.052 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.053 INFO:tasks.workunit.client.0.smithi094.stdout: "last_change": "30", 2021-07-26T06:17:14.053 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend": "0", 2021-07-26T06:17:14.053 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_prenautilus": "0", 2021-07-26T06:17:14.053 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_preluminous": "0", 2021-07-26T06:17:14.054 INFO:tasks.workunit.client.0.smithi094.stdout: "auid": 0, 2021-07-26T06:17:14.054 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_mode": "pool", 2021-07-26T06:17:14.054 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_seq": 1, 2021-07-26T06:17:14.054 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_epoch": 30, 2021-07-26T06:17:14.054 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_snaps": [ 2021-07-26T06:17:14.055 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.060 INFO:tasks.workunit.client.0.smithi094.stdout: "snapid": 1, 2021-07-26T06:17:14.060 INFO:tasks.workunit.client.0.smithi094.stdout: "stamp": "2021-07-26T06:16:08.911302+0000", 2021-07-26T06:17:14.061 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "snap1" 2021-07-26T06:17:14.061 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.061 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.061 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps": "[]", 2021-07-26T06:17:14.061 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_bytes": 0, 2021-07-26T06:17:14.062 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_objects": 0, 2021-07-26T06:17:14.062 INFO:tasks.workunit.client.0.smithi094.stdout: "tiers": [], 2021-07-26T06:17:14.062 INFO:tasks.workunit.client.0.smithi094.stdout: "tier_of": -1, 2021-07-26T06:17:14.063 INFO:tasks.workunit.client.0.smithi094.stdout: "read_tier": -1, 2021-07-26T06:17:14.063 INFO:tasks.workunit.client.0.smithi094.stdout: "write_tier": -1, 2021-07-26T06:17:14.064 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_mode": "none", 2021-07-26T06:17:14.064 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_bytes": 0, 2021-07-26T06:17:14.064 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_objects": 0, 2021-07-26T06:17:14.064 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-07-26T06:17:14.065 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-07-26T06:17:14.065 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_full_ratio_micro": 800000, 2021-07-26T06:17:14.065 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_flush_age": 0, 2021-07-26T06:17:14.065 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_evict_age": 0, 2021-07-26T06:17:14.065 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profile": "myprofile", 2021-07-26T06:17:14.066 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_params": { 2021-07-26T06:17:14.066 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "none" 2021-07-26T06:17:14.066 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.066 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_period": 0, 2021-07-26T06:17:14.067 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_count": 0, 2021-07-26T06:17:14.067 INFO:tasks.workunit.client.0.smithi094.stdout: "use_gmt_hitset": true, 2021-07-26T06:17:14.067 INFO:tasks.workunit.client.0.smithi094.stdout: "min_read_recency_for_promote": 0, 2021-07-26T06:17:14.067 INFO:tasks.workunit.client.0.smithi094.stdout: "min_write_recency_for_promote": 0, 2021-07-26T06:17:14.068 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_grade_decay_rate": 0, 2021-07-26T06:17:14.068 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_search_last_n": 0, 2021-07-26T06:17:14.068 INFO:tasks.workunit.client.0.smithi094.stdout: "grade_table": [], 2021-07-26T06:17:14.068 INFO:tasks.workunit.client.0.smithi094.stdout: "stripe_width": 8192, 2021-07-26T06:17:14.068 INFO:tasks.workunit.client.0.smithi094.stdout: "expected_num_objects": 0, 2021-07-26T06:17:14.069 INFO:tasks.workunit.client.0.smithi094.stdout: "fast_read": false, 2021-07-26T06:17:14.069 INFO:tasks.workunit.client.0.smithi094.stdout: "options": {}, 2021-07-26T06:17:14.069 INFO:tasks.workunit.client.0.smithi094.stdout: "application_metadata": {} 2021-07-26T06:17:14.069 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.069 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.070 INFO:tasks.workunit.client.0.smithi094.stdout: "osds": [ 2021-07-26T06:17:14.070 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.070 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:14.070 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "a83a0bc5-6fa4-4572-8fce-7878e9f50f58", 2021-07-26T06:17:14.071 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:17:14.072 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:17:14.072 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:17:14.072 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:17:14.072 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 5, 2021-07-26T06:17:14.073 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:17:14.073 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 37, 2021-07-26T06:17:14.073 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 41, 2021-07-26T06:17:14.073 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 31, 2021-07-26T06:17:14.074 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:17:14.074 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:14.074 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.075 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.076 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.076 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6802", 2021-07-26T06:17:14.076 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.076 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.076 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.077 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.077 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6803", 2021-07-26T06:17:14.078 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.079 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.079 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.079 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.079 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:17:14.079 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.080 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.080 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.080 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6804", 2021-07-26T06:17:14.080 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.081 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.081 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.081 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.081 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6805", 2021-07-26T06:17:14.081 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.082 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.082 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.082 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.082 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:17:14.083 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.083 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.084 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.084 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6808", 2021-07-26T06:17:14.084 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.084 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.085 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.085 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.085 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6809", 2021-07-26T06:17:14.085 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.085 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.086 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.086 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.086 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:17:14.086 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.087 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.087 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.087 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6806", 2021-07-26T06:17:14.087 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.087 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.088 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.089 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.089 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6807", 2021-07-26T06:17:14.090 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66091 2021-07-26T06:17:14.090 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.091 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.091 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.091 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6803/66091", 2021-07-26T06:17:14.091 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6805/66091", 2021-07-26T06:17:14.092 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6809/66091", 2021-07-26T06:17:14.092 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6807/66091", 2021-07-26T06:17:14.092 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:17:14.092 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:17:14.093 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:17:14.093 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.093 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.093 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.093 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:14.094 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "ffede5c8-cf1c-400b-9ab1-242a38919ce0", 2021-07-26T06:17:14.094 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 0, 2021-07-26T06:17:14.094 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:17:14.094 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:17:14.094 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:17:14.095 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 0, 2021-07-26T06:17:14.095 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 0, 2021-07-26T06:17:14.095 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 10, 2021-07-26T06:17:14.095 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 25, 2021-07-26T06:17:14.096 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 32, 2021-07-26T06:17:14.096 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:17:14.096 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:14.096 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.096 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.097 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.097 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:17:14.097 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.097 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.098 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.098 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.098 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:17:14.098 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.099 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.099 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.099 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.099 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:17:14.100 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.100 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.100 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.100 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:17:14.101 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.101 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.101 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.101 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.102 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:17:14.102 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.102 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.102 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.102 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.103 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:17:14.103 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.103 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.103 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.103 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:17:14.104 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.104 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.104 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.104 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.105 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:17:14.105 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.105 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.105 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.105 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.106 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:17:14.106 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.106 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.106 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.106 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:17:14.107 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.107 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.107 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.107 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.108 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:17:14.108 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:14.108 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.108 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.108 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.109 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/60227", 2021-07-26T06:17:14.109 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/60227", 2021-07-26T06:17:14.109 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/60227", 2021-07-26T06:17:14.111 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/60227", 2021-07-26T06:17:14.111 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:17:14.112 INFO:tasks.workunit.client.0.smithi094.stdout: "exists" 2021-07-26T06:17:14.112 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.112 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.112 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.113 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:14.113 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "93b45a7f-022d-45f0-a0fa-7fb07deeeabd", 2021-07-26T06:17:14.113 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:17:14.113 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:17:14.114 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:17:14.114 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:17:14.114 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 15, 2021-07-26T06:17:14.114 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 29, 2021-07-26T06:17:14.114 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 39, 2021-07-26T06:17:14.115 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 0, 2021-07-26T06:17:14.115 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 33, 2021-07-26T06:17:14.115 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:17:14.115 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:14.116 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.116 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.116 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.116 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:17:14.116 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.117 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.117 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.117 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.118 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:17:14.118 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.118 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.118 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.119 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.119 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:17:14.119 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.119 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.120 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.120 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:17:14.120 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.120 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.120 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.121 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.121 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:17:14.123 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.123 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.123 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.124 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.124 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:17:14.124 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.124 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.125 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.125 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:17:14.125 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.125 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.126 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.126 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.126 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:17:14.126 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.128 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.128 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.129 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.129 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:17:14.129 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:14.129 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.129 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:14.130 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:17:14.130 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.130 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.130 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 66669 2021-07-26T06:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.131 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/66669", 2021-07-26T06:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/66669", 2021-07-26T06:17:14.132 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/66669", 2021-07-26T06:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/66669", 2021-07-26T06:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:17:14.133 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:17:14.134 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.134 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.134 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.134 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_xinfo": [ 2021-07-26T06:17:14.134 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.135 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:14.135 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:16:19.962782+0000", 2021-07-26T06:17:14.135 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:17:14.135 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:17:14.136 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:17:14.136 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:17:14.136 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:14:53.481355+0000", 2021-07-26T06:17:14.140 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:17:14.140 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.140 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.140 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:14.141 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:16:20.531332+0000", 2021-07-26T06:17:14.141 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:17:14.141 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:17:14.141 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:17:14.142 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:17:14.142 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:15:05.079001+0000", 2021-07-26T06:17:14.142 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:17:14.142 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.142 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.143 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:14.143 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:16:21.163526+0000", 2021-07-26T06:17:14.143 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:17:14.143 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:17:14.144 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:17:14.144 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:17:14.144 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:15:17.547758+0000", 2021-07-26T06:17:14.144 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:17:14.144 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.145 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.145 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap": [], 2021-07-26T06:17:14.145 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap_items": [], 2021-07-26T06:17:14.145 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_temp": [], 2021-07-26T06:17:14.146 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_temp": [], 2021-07-26T06:17:14.146 INFO:tasks.workunit.client.0.smithi094.stdout: "blocklist": {}, 2021-07-26T06:17:14.146 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profiles": { 2021-07-26T06:17:14.146 INFO:tasks.workunit.client.0.smithi094.stdout: "default": { 2021-07-26T06:17:14.146 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:17:14.147 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "2", 2021-07-26T06:17:14.147 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:17:14.147 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van" 2021-07-26T06:17:14.147 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.147 INFO:tasks.workunit.client.0.smithi094.stdout: "myprofile": { 2021-07-26T06:17:14.148 INFO:tasks.workunit.client.0.smithi094.stdout: "crush-device-class": "", 2021-07-26T06:17:14.148 INFO:tasks.workunit.client.0.smithi094.stdout: "crush-failure-domain": "osd", 2021-07-26T06:17:14.148 INFO:tasks.workunit.client.0.smithi094.stdout: "crush-root": "default", 2021-07-26T06:17:14.148 INFO:tasks.workunit.client.0.smithi094.stdout: "jerasure-per-chunk-alignment": "false", 2021-07-26T06:17:14.148 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:17:14.149 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "1", 2021-07-26T06:17:14.149 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:17:14.153 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van", 2021-07-26T06:17:14.153 INFO:tasks.workunit.client.0.smithi094.stdout: "w": "8" 2021-07-26T06:17:14.154 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.154 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.154 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps_queue": [], 2021-07-26T06:17:14.155 INFO:tasks.workunit.client.0.smithi094.stdout: "new_removed_snaps": [], 2021-07-26T06:17:14.155 INFO:tasks.workunit.client.0.smithi094.stdout: "new_purged_snaps": [], 2021-07-26T06:17:14.155 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_node_flags": {}, 2021-07-26T06:17:14.155 INFO:tasks.workunit.client.0.smithi094.stdout: "device_class_flags": {}, 2021-07-26T06:17:14.156 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": { 2021-07-26T06:17:14.156 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_enabled": false, 2021-07-26T06:17:14.156 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_bucket_count": 0, 2021-07-26T06:17:14.156 INFO:tasks.workunit.client.0.smithi094.stdout: "degraded_stretch_mode": 0, 2021-07-26T06:17:14.156 INFO:tasks.workunit.client.0.smithi094.stdout: "recovering_stretch_mode": 0, 2021-07-26T06:17:14.157 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_bucket": 0 2021-07-26T06:17:14.157 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.157 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.157 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_metadata": [ 2021-07-26T06:17:14.157 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.158 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:14.158 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:17:14.158 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6804/66091,v1:127.0.0.1:6805/66091]", 2021-07-26T06:17:14.158 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:17:14.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:17:14.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:17:14.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:17:14.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:17:14.159 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:17:14.160 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:17:14.160 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:17:14.160 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:17:14.160 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-07-26T06:17:14.161 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:17:14.161 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:17:14.161 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:17:14.161 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:17:14.161 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:17:14.162 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:17:14.162 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:17:14.162 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:17:14.162 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:17:14.162 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:17:14.163 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:17:14.163 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:17:14.163 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:17:14.163 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:17:14.164 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:17:14.164 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6802/66091,v1:127.0.0.1:6803/66091]", 2021-07-26T06:17:14.164 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:17:14.164 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/66091,v1:127.0.0.1:6809/66091]", 2021-07-26T06:17:14.164 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/66091,v1:127.0.0.1:6807/66091]", 2021-07-26T06:17:14.165 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:17:14.165 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:17:14.165 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:17:14.165 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:17:14.166 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:17:14.166 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:17:14.166 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:17:14.166 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:17:14.167 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:17:14.167 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/0", 2021-07-26T06:17:14.167 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:17:14.167 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:17:14.168 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:17:14.168 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.168 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.168 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:14.168 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:17:14.169 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/60227,v1:127.0.0.1:6813/60227]", 2021-07-26T06:17:14.169 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:17:14.169 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:17:14.169 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:17:14.169 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:17:14.170 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:17:14.170 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:17:14.170 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:17:14.171 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:17:14.171 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:17:14.171 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-07-26T06:17:14.171 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:17:14.172 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:17:14.172 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:17:14.172 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:17:14.172 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:17:14.172 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:17:14.173 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:17:14.173 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:17:14.173 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:17:14.173 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:17:14.174 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:17:14.174 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:17:14.174 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:17:14.174 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:17:14.174 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:17:14.175 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/60227,v1:127.0.0.1:6811/60227]", 2021-07-26T06:17:14.175 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:17:14.175 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/60227,v1:127.0.0.1:6817/60227]", 2021-07-26T06:17:14.175 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/60227,v1:127.0.0.1:6815/60227]", 2021-07-26T06:17:14.175 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:17:14.176 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:17:14.176 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:17:14.176 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:17:14.177 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:17:14.178 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:17:14.179 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:17:14.179 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:17:14.179 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:17:14.179 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/1", 2021-07-26T06:17:14.180 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:17:14.180 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:17:14.180 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:17:14.180 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.180 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.181 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:14.181 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:17:14.181 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/66669,v1:127.0.0.1:6813/66669]", 2021-07-26T06:17:14.181 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:17:14.181 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:17:14.182 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:17:14.182 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:17:14.183 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:17:14.184 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:17:14.184 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:17:14.184 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:17:14.184 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:17:14.185 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-07-26T06:17:14.185 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:17:14.185 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:17:14.185 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:17:14.185 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:17:14.186 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:17:14.186 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:17:14.186 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:17:14.186 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:17:14.187 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:17:14.187 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:17:14.187 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:17:14.187 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:17:14.188 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:17:14.188 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:17:14.188 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:17:14.188 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/66669,v1:127.0.0.1:6811/66669]", 2021-07-26T06:17:14.189 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:17:14.189 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/66669,v1:127.0.0.1:6817/66669]", 2021-07-26T06:17:14.190 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/66669,v1:127.0.0.1:6815/66669]", 2021-07-26T06:17:14.190 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:17:14.190 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:17:14.190 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:17:14.191 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:17:14.191 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:17:14.191 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:17:14.191 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:17:14.192 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:17:14.192 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:17:14.192 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/2", 2021-07-26T06:17:14.192 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:17:14.193 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:17:14.193 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:17:14.194 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.194 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.194 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_clean_epochs": { 2021-07-26T06:17:14.195 INFO:tasks.workunit.client.0.smithi094.stdout: "min_last_epoch_clean": 0, 2021-07-26T06:17:14.195 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": { 2021-07-26T06:17:14.195 INFO:tasks.workunit.client.0.smithi094.stdout: "per_pool": [] 2021-07-26T06:17:14.196 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.196 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_epochs": [ 2021-07-26T06:17:14.198 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.198 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:14.199 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 37 2021-07-26T06:17:14.199 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.199 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.199 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:14.200 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 10 2021-07-26T06:17:14.200 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.200 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.200 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:14.200 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 39 2021-07-26T06:17:14.201 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.201 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.201 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.202 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_first_committed": 1, 2021-07-26T06:17:14.202 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_last_committed": 42, 2021-07-26T06:17:14.202 INFO:tasks.workunit.client.0.smithi094.stdout: "crushmap": { 2021-07-26T06:17:14.202 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": [ 2021-07-26T06:17:14.203 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.203 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:14.203 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.0", 2021-07-26T06:17:14.203 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:17:14.203 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.204 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.204 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:14.204 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.1", 2021-07-26T06:17:14.204 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:17:14.205 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.205 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.205 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:14.205 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.2", 2021-07-26T06:17:14.205 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:17:14.206 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.206 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.207 INFO:tasks.workunit.client.0.smithi094.stdout: "types": [ 2021-07-26T06:17:14.207 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.207 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 0, 2021-07-26T06:17:14.207 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd" 2021-07-26T06:17:14.208 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.208 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.208 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:17:14.208 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "host" 2021-07-26T06:17:14.208 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.209 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.209 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 2, 2021-07-26T06:17:14.209 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "chassis" 2021-07-26T06:17:14.209 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.209 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.210 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 3, 2021-07-26T06:17:14.210 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "rack" 2021-07-26T06:17:14.210 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.210 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.211 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 4, 2021-07-26T06:17:14.211 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "row" 2021-07-26T06:17:14.211 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.212 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.212 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 5, 2021-07-26T06:17:14.212 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pdu" 2021-07-26T06:17:14.213 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.213 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.213 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 6, 2021-07-26T06:17:14.214 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pod" 2021-07-26T06:17:14.214 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.214 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.215 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 7, 2021-07-26T06:17:14.215 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "room" 2021-07-26T06:17:14.215 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.215 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.216 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 8, 2021-07-26T06:17:14.217 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "datacenter" 2021-07-26T06:17:14.217 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.217 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.217 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 9, 2021-07-26T06:17:14.217 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "zone" 2021-07-26T06:17:14.218 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.218 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.219 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 10, 2021-07-26T06:17:14.219 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "region" 2021-07-26T06:17:14.219 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.219 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.219 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:17:14.220 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "root" 2021-07-26T06:17:14.220 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.220 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.220 INFO:tasks.workunit.client.0.smithi094.stdout: "buckets": [ 2021-07-26T06:17:14.220 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.221 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -1, 2021-07-26T06:17:14.221 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default", 2021-07-26T06:17:14.221 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:17:14.221 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:17:14.222 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:14.222 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:14.222 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:14.222 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:14.222 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.223 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:17:14.223 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:14.223 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:14.223 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.223 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.224 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.224 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.224 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -2, 2021-07-26T06:17:14.224 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default~hdd", 2021-07-26T06:17:14.225 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:17:14.225 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:17:14.225 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:14.225 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:14.225 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:14.226 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:14.226 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.226 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:17:14.227 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:14.227 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:14.227 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.227 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.228 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.228 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.228 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:17:14.228 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094", 2021-07-26T06:17:14.228 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:17:14.229 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:17:14.229 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:14.229 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:14.229 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:14.229 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:14.230 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.230 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:14.230 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:14.230 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:14.230 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.231 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.231 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:14.231 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:14.231 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:17:14.232 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.232 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.232 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:14.232 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:14.232 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:17:14.233 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.233 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.233 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.233 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.233 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:17:14.234 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094~hdd", 2021-07-26T06:17:14.234 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:17:14.234 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:17:14.234 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:14.235 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:14.235 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:14.235 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:14.235 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.235 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:14.236 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:14.236 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:14.236 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.236 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.237 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:14.237 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:14.237 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:17:14.237 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.238 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.238 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:14.238 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:14.238 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:17:14.239 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.239 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.240 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.240 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.240 INFO:tasks.workunit.client.0.smithi094.stdout: "rules": [ 2021-07-26T06:17:14.240 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.241 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 0, 2021-07-26T06:17:14.241 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "replicated_rule", 2021-07-26T06:17:14.241 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:17:14.242 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:17:14.242 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.242 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:17:14.242 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:17:14.242 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:17:14.243 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.243 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.243 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_firstn", 2021-07-26T06:17:14.243 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:17:14.243 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:17:14.244 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.244 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.244 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:17:14.244 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.245 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.245 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.245 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.245 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 1, 2021-07-26T06:17:14.245 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "test", 2021-07-26T06:17:14.246 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 3, 2021-07-26T06:17:14.246 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:17:14.246 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.246 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "set_chooseleaf_tries", 2021-07-26T06:17:14.246 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 5 2021-07-26T06:17:14.247 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.247 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.247 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "set_choose_tries", 2021-07-26T06:17:14.248 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 100 2021-07-26T06:17:14.248 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.248 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.248 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:17:14.249 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:17:14.249 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:17:14.249 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.249 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.249 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_indep", 2021-07-26T06:17:14.250 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:17:14.250 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:17:14.250 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.250 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.250 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:17:14.251 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.251 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:14.251 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.251 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.251 INFO:tasks.workunit.client.0.smithi094.stdout: "tunables": { 2021-07-26T06:17:14.252 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_tries": 0, 2021-07-26T06:17:14.252 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_fallback_tries": 0, 2021-07-26T06:17:14.252 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_total_tries": 50, 2021-07-26T06:17:14.252 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_descend_once": 1, 2021-07-26T06:17:14.253 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_vary_r": 1, 2021-07-26T06:17:14.253 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_stable": 1, 2021-07-26T06:17:14.253 INFO:tasks.workunit.client.0.smithi094.stdout: "straw_calc_version": 1, 2021-07-26T06:17:14.253 INFO:tasks.workunit.client.0.smithi094.stdout: "allowed_bucket_algs": 54, 2021-07-26T06:17:14.253 INFO:tasks.workunit.client.0.smithi094.stdout: "profile": "jewel", 2021-07-26T06:17:14.254 INFO:tasks.workunit.client.0.smithi094.stdout: "optimal_tunables": 1, 2021-07-26T06:17:14.254 INFO:tasks.workunit.client.0.smithi094.stdout: "legacy_tunables": 0, 2021-07-26T06:17:14.254 INFO:tasks.workunit.client.0.smithi094.stdout: "minimum_required_version": "jewel", 2021-07-26T06:17:14.254 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables": 1, 2021-07-26T06:17:14.254 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables2": 1, 2021-07-26T06:17:14.255 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v2_rules": 1, 2021-07-26T06:17:14.255 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables3": 1, 2021-07-26T06:17:14.255 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v3_rules": 0, 2021-07-26T06:17:14.255 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v4_buckets": 1, 2021-07-26T06:17:14.256 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables5": 1, 2021-07-26T06:17:14.256 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v5_rules": 0 2021-07-26T06:17:14.256 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.256 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_args": {} 2021-07-26T06:17:14.256 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.257 INFO:tasks.workunit.client.0.smithi094.stdout: "fsmap": { 2021-07-26T06:17:14.257 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:17:14.258 INFO:tasks.workunit.client.0.smithi094.stdout: "default_fscid": -1, 2021-07-26T06:17:14.258 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": { 2021-07-26T06:17:14.258 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": {}, 2021-07-26T06:17:14.258 INFO:tasks.workunit.client.0.smithi094.stdout: "ro_compat": {}, 2021-07-26T06:17:14.259 INFO:tasks.workunit.client.0.smithi094.stdout: "incompat": { 2021-07-26T06:17:14.259 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_1": "base v0.20", 2021-07-26T06:17:14.259 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_2": "client writeable ranges", 2021-07-26T06:17:14.259 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_3": "default file layouts on dirs", 2021-07-26T06:17:14.260 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_4": "dir inode in separate object", 2021-07-26T06:17:14.260 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_5": "mds uses versioned encoding", 2021-07-26T06:17:14.260 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_6": "dirfrag is stored in omap", 2021-07-26T06:17:14.260 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_8": "no anchor table", 2021-07-26T06:17:14.260 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_9": "file layout v2", 2021-07-26T06:17:14.261 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_10": "snaprealm v2" 2021-07-26T06:17:14.261 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.261 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.261 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_flags": { 2021-07-26T06:17:14.261 INFO:tasks.workunit.client.0.smithi094.stdout: "enable_multiple": true, 2021-07-26T06:17:14.262 INFO:tasks.workunit.client.0.smithi094.stdout: "ever_enabled_multiple": true 2021-07-26T06:17:14.262 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.262 INFO:tasks.workunit.client.0.smithi094.stdout: "standbys": [], 2021-07-26T06:17:14.262 INFO:tasks.workunit.client.0.smithi094.stdout: "filesystems": [] 2021-07-26T06:17:14.263 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.263 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_first_committed": 1, 2021-07-26T06:17:14.263 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_last_committed": 1, 2021-07-26T06:17:14.263 INFO:tasks.workunit.client.0.smithi094.stdout: "auth": { 2021-07-26T06:17:14.263 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:17:14.264 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 5, 2021-07-26T06:17:14.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_secrets": 3 2021-07-26T06:17:14.264 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:17:14.264 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_active": 1, 2021-07-26T06:17:14.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_unknown": 0, 2021-07-26T06:17:14.265 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osd": 3, 2021-07-26T06:17:14.265 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_sum": { 2021-07-26T06:17:14.265 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:17:14.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 62460, 2021-07-26T06:17:14.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 101, 2021-07-26T06:17:14.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 1, 2021-07-26T06:17:14.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 303, 2021-07-26T06:17:14.266 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:17:14.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:17:14.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 101, 2021-07-26T06:17:14.267 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:17:14.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:17:14.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 101, 2021-07-26T06:17:14.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:17:14.268 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:17:14.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:17:14.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 101, 2021-07-26T06:17:14.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 101, 2021-07-26T06:17:14.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:17:14.269 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:17:14.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:17:14.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:17:14.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:17:14.270 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:17:14.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:17:14.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:17:14.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:17:14.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:17:14.271 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:17:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:17:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:17:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:17:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:17:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:17:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:17:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:17:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:17:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:17:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:17:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:17:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:17:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:17:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:17:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:17:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:17:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:17:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 0, 2021-07-26T06:17:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 0, 2021-07-26T06:17:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:17:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:17:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 18, 2021-07-26T06:17:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 18, 2021-07-26T06:17:14.278 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 3, 2021-07-26T06:17:14.278 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 3, 2021-07-26T06:17:14.278 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 0 2021-07-26T06:17:14.278 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.279 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum": { 2021-07-26T06:17:14.279 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:17:14.279 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:17:14.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:17:14.279 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:17:14.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:17:14.280 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:17:14.280 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:17:14.280 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18564, 2021-07-26T06:17:14.280 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2148, 2021-07-26T06:17:14.281 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:17:14.281 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16381, 2021-07-26T06:17:14.281 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554236, 2021-07-26T06:17:14.281 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:17:14.282 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:17:14.282 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103537664, 2021-07-26T06:17:14.282 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:14.282 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2199552, 2021-07-26T06:17:14.282 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 1538977, 2021-07-26T06:17:14.283 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:14.283 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:14.283 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:14.283 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:17:14.283 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16775056 2021-07-26T06:17:14.284 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.284 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:17:14.284 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:17:14.284 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:17:14.284 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:17:14.285 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:17:14.285 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:17:14.285 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:17:14.285 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.285 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:17:14.286 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:17:14.286 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:17:14.286 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:17:14.286 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:17:14.287 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.287 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:17:14.287 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:17:14.287 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.287 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum_by_class": { 2021-07-26T06:17:14.288 INFO:tasks.workunit.client.0.smithi094.stdout: "hdd": { 2021-07-26T06:17:14.288 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:17:14.288 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:17:14.288 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:17:14.288 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:17:14.289 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:17:14.289 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:17:14.289 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:17:14.289 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 18564, 2021-07-26T06:17:14.289 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2148, 2021-07-26T06:17:14.290 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 2, 2021-07-26T06:17:14.290 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 16381, 2021-07-26T06:17:14.290 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314554236, 2021-07-26T06:17:14.290 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:17:14.291 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:17:14.291 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322103537664, 2021-07-26T06:17:14.291 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:14.291 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2199552, 2021-07-26T06:17:14.291 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 1538977, 2021-07-26T06:17:14.292 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:14.292 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:14.292 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:14.292 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 2160, 2021-07-26T06:17:14.292 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 16775056 2021-07-26T06:17:14.293 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.293 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:17:14.293 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:17:14.293 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:17:14.293 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:17:14.294 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:17:14.294 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:17:14.294 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:17:14.295 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.295 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:17:14.295 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:17:14.295 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:17:14.295 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:17:14.296 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:17:14.296 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.296 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:17:14.296 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:17:14.296 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.297 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.297 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_stats": [ 2021-07-26T06:17:14.297 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.297 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:17:14.298 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:17:14.298 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:17:14.298 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 62460, 2021-07-26T06:17:14.298 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 101, 2021-07-26T06:17:14.298 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 1, 2021-07-26T06:17:14.299 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 303, 2021-07-26T06:17:14.299 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:17:14.299 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:17:14.299 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 101, 2021-07-26T06:17:14.299 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:17:14.300 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:17:14.300 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 101, 2021-07-26T06:17:14.300 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:17:14.300 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:17:14.300 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:17:14.301 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 101, 2021-07-26T06:17:14.301 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 101, 2021-07-26T06:17:14.301 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:17:14.301 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:17:14.302 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:17:14.302 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:17:14.302 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:17:14.302 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:17:14.302 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:17:14.303 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:17:14.303 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:17:14.303 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:17:14.303 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:17:14.303 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:17:14.304 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:17:14.304 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:17:14.304 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:17:14.304 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:17:14.304 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:17:14.305 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:17:14.305 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:17:14.305 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:17:14.305 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:17:14.305 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:17:14.306 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:17:14.306 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:17:14.306 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:17:14.306 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.307 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:17:14.307 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:17:14.307 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:17:14.307 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:14.307 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 1241088, 2021-07-26T06:17:14.308 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 1241088, 2021-07-26T06:17:14.308 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:14.308 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:14.308 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:14.308 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:17:14.309 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:17:14.309 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.309 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 18, 2021-07-26T06:17:14.309 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 18, 2021-07-26T06:17:14.309 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 3, 2021-07-26T06:17:14.310 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 3, 2021-07-26T06:17:14.310 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 3 2021-07-26T06:17:14.310 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.310 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.311 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_stats": [ 2021-07-26T06:17:14.311 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.311 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:14.311 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 158913789964 2021-07-26T06:17:14.311 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.312 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.312 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:14.312 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 42949672983 2021-07-26T06:17:14.312 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.312 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.313 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:14.313 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 167503724555 2021-07-26T06:17:14.313 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.313 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.314 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_state": [ 2021-07-26T06:17:14.314 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.314 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized+degraded", 2021-07-26T06:17:14.314 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 1 2021-07-26T06:17:14.314 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.315 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.315 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_osd": [ 2021-07-26T06:17:14.315 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.315 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2147483647, 2021-07-26T06:17:14.315 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:17:14.316 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:17:14.316 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:14.316 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.316 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.316 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:14.317 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:17:14.317 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:17:14.317 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:14.317 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.317 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.318 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:14.318 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 1, 2021-07-26T06:17:14.318 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:17:14.318 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:14.319 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.319 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.319 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:14.319 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:17:14.319 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 0, 2021-07-26T06:17:14.320 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:14.320 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.320 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.321 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": [ 2021-07-26T06:17:14.321 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:14.321 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:17:14.321 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": {} 2021-07-26T06:17:14.322 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.322 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:14.322 INFO:tasks.workunit.client.0.smithi094.stdout: "servicemap": { 2021-07-26T06:17:14.322 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 5, 2021-07-26T06:17:14.323 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:16:18.814309+0000", 2021-07-26T06:17:14.323 INFO:tasks.workunit.client.0.smithi094.stdout: "services": {} 2021-07-26T06:17:14.323 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:14.323 INFO:tasks.workunit.client.0.smithi094.stdout: "paxos": { 2021-07-26T06:17:14.323 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:17:14.324 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 424, 2021-07-26T06:17:14.324 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pn": 0, 2021-07-26T06:17:14.324 INFO:tasks.workunit.client.0.smithi094.stdout: "accepted_pn": 0 2021-07-26T06:17:14.324 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:14.324 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:17:14.327 INFO:tasks.workunit.client.0.smithi094.stderr:report 2682747688 2021-07-26T06:17:14.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: return 1 2021-07-26T06:17:14.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:17:14.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2021-07-26T06:17:14.328 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:17:14.328 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:17:14.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:17:14.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:17:14.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:17:14.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:17:14.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2021-07-26T06:17:14.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-07-26T06:17:14.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:17:14.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=0 2021-07-26T06:17:14.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:17:14.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2021-07-26T06:17:14.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2021-07-26T06:17:15.396 INFO:tasks.workunit.client.0.smithi094.stderr:PG '1.0' not found 2021-07-26T06:17:15.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nodown: return 1 2021-07-26T06:17:15.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-07-26T06:17:15.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:17:15.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:17:15.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:17:15.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:17:15.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/0 2021-07-26T06:17:15.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:17:15.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:17:15.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:17:15.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:17:15.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-07-26T06:17:15.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-07-26T06:17:15.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:17:15.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:17:15.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:17:15.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:17:15.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:17:15.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:17:15.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:17:15.942 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:17:15.942 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:17:15.942 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:17:15.942 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:17:15.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:17:15.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:17:15.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:17:15.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:17:15.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:17:15.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:17:15.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:17:15.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:17:15.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:17:15.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/0 2021-07-26T06:17:15.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:17:15.948 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:17:15.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:17:15.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/0/whoami 2021-07-26T06:17:15.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:17:15.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:17:15.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:17:15.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:17:15.980 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:15.971+0000 7effb5713f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:15.982 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:15.975+0000 7effb5713f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:15.986 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:15.975+0000 7effb5713f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:16.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:17:16.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:17:16.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:17:16.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:17:16.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:17:16.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:17:16.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:17:16.288 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:17:16.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:17:16.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:17:16.524 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:16.515+0000 7effb5713f40 -1 Falling back to public interface 2021-07-26T06:17:16.713 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 37 up_thru 41 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/66091,v1:127.0.0.1:6803/66091] [v2:127.0.0.1:6804/66091,v1:127.0.0.1:6805/66091] exists,up a83a0bc5-6fa4-4572-8fce-7878e9f50f58 2021-07-26T06:17:16.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:17:16.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:17:16.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:17:16.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:17:16.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2021-07-26T06:17:16.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:17:16.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:17:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:17:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:17:16.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:17:16.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:17:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2021-07-26T06:17:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-07-26T06:17:16.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:17:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=2 2021-07-26T06:17:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:17:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-07-26T06:17:16.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2021-07-26T06:17:17.932 INFO:tasks.workunit.client.0.smithi094.stderr:PG '1.0' not found 2021-07-26T06:17:18.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nodown: return 1 2021-07-26T06:17:18.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-07-26T06:17:18.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:17:18.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:17:18.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:17:18.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:17:18.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/2 2021-07-26T06:17:18.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:17:18.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:17:18.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:17:18.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:17:18.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-07-26T06:17:18.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-07-26T06:17:18.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:17:18.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:17:18.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:17:18.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:17:18.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:17:18.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:17:18.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:17:18.481 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:17:18.482 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:17:18.482 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:17:18.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:17:18.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:17:18.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:17:18.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:17:18.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:17:18.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:17:18.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:17:18.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:17:18.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:17:18.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:17:18.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/2 2021-07-26T06:17:18.487 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:17:18.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:17:18.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:17:18.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/2/whoami 2021-07-26T06:17:18.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:17:18.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:17:18.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:17:18.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:17:18.517 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:18.506+0000 7fb950c82f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:18.520 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:18.510+0000 7fb950c82f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:18.526 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:18.510+0000 7fb950c82f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:18.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:17:18.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:17:18.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:17:18.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:17:18.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:17:18.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:17:18.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:17:18.878 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:17:18.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:17:18.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:17:19.061 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:19.050+0000 7fb950c82f40 -1 Falling back to public interface 2021-07-26T06:17:19.290 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:19.278+0000 7effb5713f40 -1 osd.0 42 log_to_monitors {default=true} 2021-07-26T06:17:19.373 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/66669,v1:127.0.0.1:6811/66669] [v2:127.0.0.1:6812/66669,v1:127.0.0.1:6813/66669] exists,up 93b45a7f-022d-45f0-a0fa-7fb07deeeabd 2021-07-26T06:17:19.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:17:19.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:17:19.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:17:19.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2021-07-26T06:17:19.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-07-26T06:17:19.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: _objectstore_tool_nodown: shift 2021-07-26T06:17:19.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: local id=1 2021-07-26T06:17:19.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: shift 2021-07-26T06:17:19.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-07-26T06:17:19.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2021-07-26T06:17:21.624 INFO:tasks.workunit.client.0.smithi094.stderr:PG '1.0' not found 2021-07-26T06:17:21.901 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:21.890+0000 7fb950c82f40 -1 osd.2 42 log_to_monitors {default=true} 2021-07-26T06:17:21.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nodown: return 1 2021-07-26T06:17:21.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-07-26T06:17:21.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2021-07-26T06:17:21.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:17:21.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:17:21.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:17:21.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:17:21.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 20 .1 2021-07-26T06:17:21.934 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:17:21.934 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:17:21.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:17:21.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:17:21.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:17:22.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:17:22.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:17:22.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:17:22.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:17:22.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:17:22.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:17:22.496 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:17:22.497 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:17:22.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:17:22.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:17:22.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:17:22.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=193273528323 2021-07-26T06:17:22.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 193273528323 2021-07-26T06:17:22.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-193273528323' 2021-07-26T06:17:22.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:17:22.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:17:22.922 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-07-26T06:17:22.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T06:17:22.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T06:17:22.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T06:17:22.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:17:22.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:17:23.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=210453397507 2021-07-26T06:17:23.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 210453397507 2021-07-26T06:17:23.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-193273528323 2-210453397507' 2021-07-26T06:17:23.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:17:23.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-193273528323 2021-07-26T06:17:23.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:17:23.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:17:23.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-193273528323 2021-07-26T06:17:23.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:17:23.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=193273528323 2021-07-26T06:17:23.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 193273528323' 2021-07-26T06:17:23.121 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 193273528323 2021-07-26T06:17:23.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:17:23.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 193273528323 -lt 193273528323 2021-07-26T06:17:23.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:17:23.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-210453397507 2021-07-26T06:17:23.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:17:23.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:17:23.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-210453397507 2021-07-26T06:17:23.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:17:23.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=210453397507 2021-07-26T06:17:23.532 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 210453397507 2021-07-26T06:17:23.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2021-07-26T06:17:23.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:17:23.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 210453397507 2021-07-26T06:17:23.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:17:24.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:17:24.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:17:25.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 210453397507 -lt 210453397507 2021-07-26T06:17:25.336 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:17:25.336 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:25.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:25.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:17:25.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:25.814 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:25.814 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:25.814 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:25.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:25.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:25.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:26.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:26.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:26.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:26.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:26.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:26.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:17:26.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:17:26.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:17:26.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:26.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:17:26.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:26.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:26.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:26.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:26.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:26.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:26.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:26.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:27.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:27.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:27.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:27.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:27.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:27.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:27.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:27.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:27.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:27.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:27.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:27.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:27.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:28.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:28.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:28.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 1 >= 8 )) 2021-07-26T06:17:28.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:28.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.2 2021-07-26T06:17:28.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:28.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:28.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:28.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:28.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:28.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:28.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:28.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:28.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:28.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:28.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:28.697 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:29.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:29.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:29.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:29.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:29.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:29.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:29.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:29.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:29.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:29.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:29.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:29.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 2 >= 8 )) 2021-07-26T06:17:29.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:29.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.4 2021-07-26T06:17:30.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:30.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:30.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:30.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:30.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:30.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:30.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:30.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:30.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:30.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:30.449 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:30.449 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:30.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:30.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:30.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:30.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:30.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:30.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:30.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:30.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:30.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:31.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:31.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:31.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 3 >= 8 )) 2021-07-26T06:17:31.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:31.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.8 2021-07-26T06:17:32.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:32.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:32.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:32.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:32.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:32.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:32.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:32.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:32.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:32.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:32.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:32.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:33.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:33.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:33.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:33.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:33.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:33.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:33.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:33.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 4 >= 8 )) 2021-07-26T06:17:33.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:33.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 1.6 2021-07-26T06:17:35.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:35.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:35.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:35.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:35.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:35.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:35.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:35.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:35.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:35.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:35.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:35.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:35.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:35.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:35.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:35.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:35.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:35.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:35.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:35.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:35.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:36.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:36.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:36.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 5 >= 8 )) 2021-07-26T06:17:36.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:36.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 3.2 2021-07-26T06:17:39.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:39.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:39.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:39.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:39.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:39.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:39.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:39.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:39.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:39.974 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:39.974 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:39.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:40.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:40.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:40.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:40.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:40.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:40.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:40.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:40.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:40.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:40.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:40.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:40.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 6 >= 8 )) 2021-07-26T06:17:40.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:40.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 6.4 2021-07-26T06:17:47.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:47.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:47.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:47.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:47.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:47.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:47.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:47.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:47.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:47.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:47.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:47.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:48.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:48.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:48.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:48.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:48.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:48.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:48.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:48.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:48.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:48.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:48.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:48.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 7 >= 8 )) 2021-07-26T06:17:48.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:17:48.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 7.3 2021-07-26T06:17:55.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:17:55.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:17:55.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:17:55.979 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:17:55.979 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:17:55.979 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:17:55.979 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:17:55.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:17:56.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:17:56.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:17:56.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:17:56.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:17:56.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T06:17:56.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:17:56.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:17:56.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:17:56.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:17:56.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:17:56.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:17:56.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:17:56.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:17:57.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=null 2021-07-26T06:17:57.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test null '!=' null 2021-07-26T06:17:57.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: (( 8 >= 8 )) 2021-07-26T06:17:57.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph report 2021-07-26T06:17:57.752 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:17:57.752 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_fingerprint": "4722d731-e412-4982-951f-8bd4beca4e17", 2021-07-26T06:17:57.753 INFO:tasks.workunit.client.0.smithi094.stdout: "version": "17.0.0-6358-g197e564f", 2021-07-26T06:17:57.753 INFO:tasks.workunit.client.0.smithi094.stdout: "commit": "197e564fdaee12596de1dc8f3253d1704ae66d5d", 2021-07-26T06:17:57.753 INFO:tasks.workunit.client.0.smithi094.stdout: "timestamp": "2021-07-26T06:17:57.741521+0000", 2021-07-26T06:17:57.753 INFO:tasks.workunit.client.0.smithi094.stdout: "tag": "", 2021-07-26T06:17:57.754 INFO:tasks.workunit.client.0.smithi094.stdout: "health": { 2021-07-26T06:17:57.754 INFO:tasks.workunit.client.0.smithi094.stdout: "status": "HEALTH_WARN", 2021-07-26T06:17:57.754 INFO:tasks.workunit.client.0.smithi094.stdout: "checks": { 2021-07-26T06:17:57.754 INFO:tasks.workunit.client.0.smithi094.stdout: "OSDMAP_FLAGS": { 2021-07-26T06:17:57.755 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:57.755 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:57.755 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "nodown,noout,noin flag(s) set", 2021-07-26T06:17:57.755 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 17 2021-07-26T06:17:57.755 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.756 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [], 2021-07-26T06:17:57.756 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:57.756 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.756 INFO:tasks.workunit.client.0.smithi094.stdout: "OSD_DOWN": { 2021-07-26T06:17:57.757 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:57.757 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:57.757 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 osds down", 2021-07-26T06:17:57.757 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:17:57.757 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.758 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:17:57.758 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.758 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "osd.1 (root=default,host=smithi094) is down" 2021-07-26T06:17:57.758 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.759 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.759 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:57.759 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.759 INFO:tasks.workunit.client.0.smithi094.stdout: "PG_DEGRADED": { 2021-07-26T06:17:57.759 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:57.760 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:57.760 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-07-26T06:17:57.760 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:17:57.760 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.761 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:17:57.761 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.761 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2021-07-26T06:17:57.761 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.761 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.762 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:57.762 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.762 INFO:tasks.workunit.client.0.smithi094.stdout: "POOL_APP_NOT_ENABLED": { 2021-07-26T06:17:57.762 INFO:tasks.workunit.client.0.smithi094.stdout: "severity": "HEALTH_WARN", 2021-07-26T06:17:57.763 INFO:tasks.workunit.client.0.smithi094.stdout: "summary": { 2021-07-26T06:17:57.763 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "1 pool(s) do not have an application enabled", 2021-07-26T06:17:57.763 INFO:tasks.workunit.client.0.smithi094.stdout: "count": 1 2021-07-26T06:17:57.763 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.763 INFO:tasks.workunit.client.0.smithi094.stdout: "detail": [ 2021-07-26T06:17:57.764 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.764 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "application not enabled on pool 'test'" 2021-07-26T06:17:57.764 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.764 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.765 INFO:tasks.workunit.client.0.smithi094.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-07-26T06:17:57.765 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.765 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.765 INFO:tasks.workunit.client.0.smithi094.stdout: "muted": false 2021-07-26T06:17:57.765 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.766 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.766 INFO:tasks.workunit.client.0.smithi094.stdout: "mutes": [] 2021-07-26T06:17:57.766 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.766 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_first_committed": 1, 2021-07-26T06:17:57.766 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap_last_committed": 1, 2021-07-26T06:17:57.767 INFO:tasks.workunit.client.0.smithi094.stdout: "monmap": { 2021-07-26T06:17:57.767 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:17:57.767 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:17:57.768 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:14:41.137869Z", 2021-07-26T06:17:57.768 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:14:41.137869Z", 2021-07-26T06:17:57.768 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release": 17, 2021-07-26T06:17:57.769 INFO:tasks.workunit.client.0.smithi094.stdout: "min_mon_release_name": "quincy", 2021-07-26T06:17:57.769 INFO:tasks.workunit.client.0.smithi094.stdout: "election_strategy": 1, 2021-07-26T06:17:57.769 INFO:tasks.workunit.client.0.smithi094.stdout: "disallowed_leaders: ": "", 2021-07-26T06:17:57.769 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": false, 2021-07-26T06:17:57.769 INFO:tasks.workunit.client.0.smithi094.stdout: "features": { 2021-07-26T06:17:57.770 INFO:tasks.workunit.client.0.smithi094.stdout: "persistent": [ 2021-07-26T06:17:57.770 INFO:tasks.workunit.client.0.smithi094.stdout: "kraken", 2021-07-26T06:17:57.770 INFO:tasks.workunit.client.0.smithi094.stdout: "luminous", 2021-07-26T06:17:57.770 INFO:tasks.workunit.client.0.smithi094.stdout: "mimic", 2021-07-26T06:17:57.771 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap-prune", 2021-07-26T06:17:57.771 INFO:tasks.workunit.client.0.smithi094.stdout: "nautilus", 2021-07-26T06:17:57.771 INFO:tasks.workunit.client.0.smithi094.stdout: "octopus", 2021-07-26T06:17:57.771 INFO:tasks.workunit.client.0.smithi094.stdout: "pacific", 2021-07-26T06:17:57.771 INFO:tasks.workunit.client.0.smithi094.stdout: "elector-pinging", 2021-07-26T06:17:57.772 INFO:tasks.workunit.client.0.smithi094.stdout: "quincy" 2021-07-26T06:17:57.772 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.772 INFO:tasks.workunit.client.0.smithi094.stdout: "optional": [] 2021-07-26T06:17:57.773 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.773 INFO:tasks.workunit.client.0.smithi094.stdout: "mons": [ 2021-07-26T06:17:57.773 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.773 INFO:tasks.workunit.client.0.smithi094.stdout: "rank": 0, 2021-07-26T06:17:57.774 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "a", 2021-07-26T06:17:57.774 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:57.774 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.774 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.774 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.775 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115", 2021-07-26T06:17:57.775 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 0 2021-07-26T06:17:57.775 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.775 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.775 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.776 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:7115/0", 2021-07-26T06:17:57.776 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:7115/0", 2021-07-26T06:17:57.776 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 0, 2021-07-26T06:17:57.777 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 0, 2021-07-26T06:17:57.777 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_location": "{}" 2021-07-26T06:17:57.777 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.777 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.778 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.778 INFO:tasks.workunit.client.0.smithi094.stdout: "quorum": [ 2021-07-26T06:17:57.778 INFO:tasks.workunit.client.0.smithi094.stdout: 0 2021-07-26T06:17:57.778 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.778 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap": { 2021-07-26T06:17:57.779 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 50, 2021-07-26T06:17:57.779 INFO:tasks.workunit.client.0.smithi094.stdout: "fsid": "01dff7b3-ca67-411f-8a1d-dda47809b70a", 2021-07-26T06:17:57.779 INFO:tasks.workunit.client.0.smithi094.stdout: "created": "2021-07-26T06:14:42.057212+0000", 2021-07-26T06:17:57.779 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:17:22.974159+0000", 2021-07-26T06:17:57.780 INFO:tasks.workunit.client.0.smithi094.stdout: "last_up_change": "2021-07-26T06:17:22.504456+0000", 2021-07-26T06:17:57.780 INFO:tasks.workunit.client.0.smithi094.stdout: "last_in_change": "2021-07-26T06:16:33.869345+0000", 2021-07-26T06:17:57.780 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-07-26T06:17:57.781 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_num": 5800384, 2021-07-26T06:17:57.782 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_set": [ 2021-07-26T06:17:57.782 INFO:tasks.workunit.client.0.smithi094.stdout: "nodown", 2021-07-26T06:17:57.783 INFO:tasks.workunit.client.0.smithi094.stdout: "noin", 2021-07-26T06:17:57.783 INFO:tasks.workunit.client.0.smithi094.stdout: "noout", 2021-07-26T06:17:57.783 INFO:tasks.workunit.client.0.smithi094.stdout: "pglog_hardlimit", 2021-07-26T06:17:57.783 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snapdirs", 2021-07-26T06:17:57.783 INFO:tasks.workunit.client.0.smithi094.stdout: "recovery_deletes", 2021-07-26T06:17:57.784 INFO:tasks.workunit.client.0.smithi094.stdout: "sortbitwise" 2021-07-26T06:17:57.784 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.784 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_version": 12, 2021-07-26T06:17:57.784 INFO:tasks.workunit.client.0.smithi094.stdout: "full_ratio": 0.99000000953674316, 2021-07-26T06:17:57.785 INFO:tasks.workunit.client.0.smithi094.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-07-26T06:17:57.785 INFO:tasks.workunit.client.0.smithi094.stdout: "nearfull_ratio": 0.85000002384185791, 2021-07-26T06:17:57.785 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_snapshot": "", 2021-07-26T06:17:57.785 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_max": 1, 2021-07-26T06:17:57.785 INFO:tasks.workunit.client.0.smithi094.stdout: "max_osd": 3, 2021-07-26T06:17:57.786 INFO:tasks.workunit.client.0.smithi094.stdout: "require_min_compat_client": "luminous", 2021-07-26T06:17:57.786 INFO:tasks.workunit.client.0.smithi094.stdout: "min_compat_client": "jewel", 2021-07-26T06:17:57.786 INFO:tasks.workunit.client.0.smithi094.stdout: "require_osd_release": "quincy", 2021-07-26T06:17:57.786 INFO:tasks.workunit.client.0.smithi094.stdout: "pools": [ 2021-07-26T06:17:57.787 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.787 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:17:57.787 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_name": "test", 2021-07-26T06:17:57.787 INFO:tasks.workunit.client.0.smithi094.stdout: "create_time": "2021-07-26T06:15:21.917164+0000", 2021-07-26T06:17:57.787 INFO:tasks.workunit.client.0.smithi094.stdout: "flags": 16389, 2021-07-26T06:17:57.788 INFO:tasks.workunit.client.0.smithi094.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-07-26T06:17:57.788 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 3, 2021-07-26T06:17:57.788 INFO:tasks.workunit.client.0.smithi094.stdout: "size": 3, 2021-07-26T06:17:57.788 INFO:tasks.workunit.client.0.smithi094.stdout: "min_size": 2, 2021-07-26T06:17:57.788 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_rule": 1, 2021-07-26T06:17:57.789 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_count": 0, 2021-07-26T06:17:57.789 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_target": 0, 2021-07-26T06:17:57.789 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_barrier": 0, 2021-07-26T06:17:57.789 INFO:tasks.workunit.client.0.smithi094.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-07-26T06:17:57.790 INFO:tasks.workunit.client.0.smithi094.stdout: "object_hash": 2, 2021-07-26T06:17:57.790 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_autoscale_mode": "off", 2021-07-26T06:17:57.790 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num": 1, 2021-07-26T06:17:57.790 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num": 1, 2021-07-26T06:17:57.791 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_placement_num_target": 1, 2021-07-26T06:17:57.791 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_target": 1, 2021-07-26T06:17:57.791 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_num_pending": 1, 2021-07-26T06:17:57.791 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pg_merge_meta": { 2021-07-26T06:17:57.791 INFO:tasks.workunit.client.0.smithi094.stdout: "source_pgid": "0.0", 2021-07-26T06:17:57.792 INFO:tasks.workunit.client.0.smithi094.stdout: "ready_epoch": 0, 2021-07-26T06:17:57.792 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_started": 0, 2021-07-26T06:17:57.792 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": 0, 2021-07-26T06:17:57.792 INFO:tasks.workunit.client.0.smithi094.stdout: "source_version": "0'0", 2021-07-26T06:17:57.792 INFO:tasks.workunit.client.0.smithi094.stdout: "target_version": "0'0" 2021-07-26T06:17:57.793 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.793 INFO:tasks.workunit.client.0.smithi094.stdout: "last_change": "30", 2021-07-26T06:17:57.793 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend": "0", 2021-07-26T06:17:57.793 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_prenautilus": "0", 2021-07-26T06:17:57.794 INFO:tasks.workunit.client.0.smithi094.stdout: "last_force_op_resend_preluminous": "0", 2021-07-26T06:17:57.794 INFO:tasks.workunit.client.0.smithi094.stdout: "auid": 0, 2021-07-26T06:17:57.794 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_mode": "pool", 2021-07-26T06:17:57.794 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_seq": 1, 2021-07-26T06:17:57.794 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_epoch": 30, 2021-07-26T06:17:57.795 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_snaps": [ 2021-07-26T06:17:57.795 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.796 INFO:tasks.workunit.client.0.smithi094.stdout: "snapid": 1, 2021-07-26T06:17:57.796 INFO:tasks.workunit.client.0.smithi094.stdout: "stamp": "2021-07-26T06:16:08.911302+0000", 2021-07-26T06:17:57.796 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "snap1" 2021-07-26T06:17:57.797 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.797 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.797 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps": "[]", 2021-07-26T06:17:57.797 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_bytes": 0, 2021-07-26T06:17:57.797 INFO:tasks.workunit.client.0.smithi094.stdout: "quota_max_objects": 0, 2021-07-26T06:17:57.798 INFO:tasks.workunit.client.0.smithi094.stdout: "tiers": [], 2021-07-26T06:17:57.798 INFO:tasks.workunit.client.0.smithi094.stdout: "tier_of": -1, 2021-07-26T06:17:57.798 INFO:tasks.workunit.client.0.smithi094.stdout: "read_tier": -1, 2021-07-26T06:17:57.798 INFO:tasks.workunit.client.0.smithi094.stdout: "write_tier": -1, 2021-07-26T06:17:57.799 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_mode": "none", 2021-07-26T06:17:57.799 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_bytes": 0, 2021-07-26T06:17:57.799 INFO:tasks.workunit.client.0.smithi094.stdout: "target_max_objects": 0, 2021-07-26T06:17:57.800 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-07-26T06:17:57.800 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-07-26T06:17:57.801 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_target_full_ratio_micro": 800000, 2021-07-26T06:17:57.801 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_flush_age": 0, 2021-07-26T06:17:57.801 INFO:tasks.workunit.client.0.smithi094.stdout: "cache_min_evict_age": 0, 2021-07-26T06:17:57.802 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profile": "myprofile", 2021-07-26T06:17:57.802 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_params": { 2021-07-26T06:17:57.802 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "none" 2021-07-26T06:17:57.802 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.803 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_period": 0, 2021-07-26T06:17:57.808 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_count": 0, 2021-07-26T06:17:57.808 INFO:tasks.workunit.client.0.smithi094.stdout: "use_gmt_hitset": true, 2021-07-26T06:17:57.808 INFO:tasks.workunit.client.0.smithi094.stdout: "min_read_recency_for_promote": 0, 2021-07-26T06:17:57.808 INFO:tasks.workunit.client.0.smithi094.stdout: "min_write_recency_for_promote": 0, 2021-07-26T06:17:57.809 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_grade_decay_rate": 0, 2021-07-26T06:17:57.809 INFO:tasks.workunit.client.0.smithi094.stdout: "hit_set_search_last_n": 0, 2021-07-26T06:17:57.809 INFO:tasks.workunit.client.0.smithi094.stdout: "grade_table": [], 2021-07-26T06:17:57.809 INFO:tasks.workunit.client.0.smithi094.stdout: "stripe_width": 8192, 2021-07-26T06:17:57.809 INFO:tasks.workunit.client.0.smithi094.stdout: "expected_num_objects": 0, 2021-07-26T06:17:57.810 INFO:tasks.workunit.client.0.smithi094.stdout: "fast_read": false, 2021-07-26T06:17:57.810 INFO:tasks.workunit.client.0.smithi094.stdout: "options": {}, 2021-07-26T06:17:57.810 INFO:tasks.workunit.client.0.smithi094.stdout: "application_metadata": {} 2021-07-26T06:17:57.810 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.811 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.811 INFO:tasks.workunit.client.0.smithi094.stdout: "osds": [ 2021-07-26T06:17:57.811 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.811 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:57.811 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "a83a0bc5-6fa4-4572-8fce-7878e9f50f58", 2021-07-26T06:17:57.812 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:17:57.812 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:17:57.812 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:17:57.812 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:17:57.813 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 37, 2021-07-26T06:17:57.813 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 42, 2021-07-26T06:17:57.813 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 45, 2021-07-26T06:17:57.813 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 49, 2021-07-26T06:17:57.814 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 44, 2021-07-26T06:17:57.814 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:17:57.815 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:57.815 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.815 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.815 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.815 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6802", 2021-07-26T06:17:57.816 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.816 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.816 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.816 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.816 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6803", 2021-07-26T06:17:57.817 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.817 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.817 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.817 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.818 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:17:57.818 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.818 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.818 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.819 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6804", 2021-07-26T06:17:57.819 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.819 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.819 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.819 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.820 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6805", 2021-07-26T06:17:57.820 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.820 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.820 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.820 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.821 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:17:57.821 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.821 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.821 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.822 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6808", 2021-07-26T06:17:57.822 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.822 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.822 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.823 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.823 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6809", 2021-07-26T06:17:57.823 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.823 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.823 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.824 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.824 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:17:57.824 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.824 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.824 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.825 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6806", 2021-07-26T06:17:57.825 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.825 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.826 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.826 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.826 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6807", 2021-07-26T06:17:57.826 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 68688 2021-07-26T06:17:57.828 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.828 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.828 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.829 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6803/68688", 2021-07-26T06:17:57.829 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6805/68688", 2021-07-26T06:17:57.829 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6809/68688", 2021-07-26T06:17:57.829 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6807/68688", 2021-07-26T06:17:57.830 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:17:57.830 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:17:57.830 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:17:57.830 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.831 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.831 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.831 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:57.831 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "ffede5c8-cf1c-400b-9ab1-242a38919ce0", 2021-07-26T06:17:57.831 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 0, 2021-07-26T06:17:57.832 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:17:57.832 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:17:57.833 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:17:57.834 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 0, 2021-07-26T06:17:57.834 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 0, 2021-07-26T06:17:57.834 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 10, 2021-07-26T06:17:57.834 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 25, 2021-07-26T06:17:57.835 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 32, 2021-07-26T06:17:57.835 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:17:57.835 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:57.836 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.836 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.836 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.836 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:17:57.837 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.837 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.837 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.837 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.838 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:17:57.838 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.838 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.839 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.839 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.839 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:17:57.839 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.840 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.840 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.841 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:17:57.841 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.841 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.841 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.842 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.843 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:17:57.843 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.845 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.845 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.845 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.846 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:17:57.846 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.846 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.846 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.847 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:17:57.847 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.847 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.847 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.848 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.848 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:17:57.848 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.848 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.848 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.849 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.849 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:17:57.849 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.849 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.849 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.850 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:17:57.850 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.850 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.850 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.851 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.851 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:17:57.851 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 60227 2021-07-26T06:17:57.852 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.852 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.853 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.854 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/60227", 2021-07-26T06:17:57.854 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/60227", 2021-07-26T06:17:57.854 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/60227", 2021-07-26T06:17:57.854 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/60227", 2021-07-26T06:17:57.855 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:17:57.855 INFO:tasks.workunit.client.0.smithi094.stdout: "exists" 2021-07-26T06:17:57.855 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.856 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.856 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.856 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:57.856 INFO:tasks.workunit.client.0.smithi094.stdout: "uuid": "93b45a7f-022d-45f0-a0fa-7fb07deeeabd", 2021-07-26T06:17:57.857 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 1, 2021-07-26T06:17:57.857 INFO:tasks.workunit.client.0.smithi094.stdout: "in": 1, 2021-07-26T06:17:57.857 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 1, 2021-07-26T06:17:57.857 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_affinity": 1, 2021-07-26T06:17:57.857 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_begin": 39, 2021-07-26T06:17:57.858 INFO:tasks.workunit.client.0.smithi094.stdout: "last_clean_end": 42, 2021-07-26T06:17:57.858 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 49, 2021-07-26T06:17:57.858 INFO:tasks.workunit.client.0.smithi094.stdout: "up_thru": 0, 2021-07-26T06:17:57.858 INFO:tasks.workunit.client.0.smithi094.stdout: "down_at": 48, 2021-07-26T06:17:57.858 INFO:tasks.workunit.client.0.smithi094.stdout: "lost_at": 0, 2021-07-26T06:17:57.859 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addrs": { 2021-07-26T06:17:57.859 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.859 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.859 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.860 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6810", 2021-07-26T06:17:57.860 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.860 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.860 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.860 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.861 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6811", 2021-07-26T06:17:57.861 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.861 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.861 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.861 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.862 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addrs": { 2021-07-26T06:17:57.862 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.863 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.863 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.863 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6812", 2021-07-26T06:17:57.864 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.864 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.864 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.865 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.865 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6813", 2021-07-26T06:17:57.866 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.867 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.867 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.868 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.869 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addrs": { 2021-07-26T06:17:57.869 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.869 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.871 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.871 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6816", 2021-07-26T06:17:57.871 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.871 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.872 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.872 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.872 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6817", 2021-07-26T06:17:57.873 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.873 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.873 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.873 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.874 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addrs": { 2021-07-26T06:17:57.874 INFO:tasks.workunit.client.0.smithi094.stdout: "addrvec": [ 2021-07-26T06:17:57.874 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.874 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v2", 2021-07-26T06:17:57.876 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6814", 2021-07-26T06:17:57.876 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.876 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.877 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.877 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "v1", 2021-07-26T06:17:57.877 INFO:tasks.workunit.client.0.smithi094.stdout: "addr": "127.0.0.1:6815", 2021-07-26T06:17:57.877 INFO:tasks.workunit.client.0.smithi094.stdout: "nonce": 69366 2021-07-26T06:17:57.877 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.878 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.878 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.878 INFO:tasks.workunit.client.0.smithi094.stdout: "public_addr": "127.0.0.1:6811/69366", 2021-07-26T06:17:57.879 INFO:tasks.workunit.client.0.smithi094.stdout: "cluster_addr": "127.0.0.1:6813/69366", 2021-07-26T06:17:57.879 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_back_addr": "127.0.0.1:6817/69366", 2021-07-26T06:17:57.880 INFO:tasks.workunit.client.0.smithi094.stdout: "heartbeat_front_addr": "127.0.0.1:6815/69366", 2021-07-26T06:17:57.880 INFO:tasks.workunit.client.0.smithi094.stdout: "state": [ 2021-07-26T06:17:57.880 INFO:tasks.workunit.client.0.smithi094.stdout: "exists", 2021-07-26T06:17:57.880 INFO:tasks.workunit.client.0.smithi094.stdout: "up" 2021-07-26T06:17:57.880 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.881 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.881 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.881 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_xinfo": [ 2021-07-26T06:17:57.881 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.882 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:57.882 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:17:19.698370+0000", 2021-07-26T06:17:57.882 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:17:57.882 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:17:57.882 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:17:57.883 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:17:57.883 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:14:53.481355+0000", 2021-07-26T06:17:57.883 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:17:57.883 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.883 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.884 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:57.884 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:16:20.531332+0000", 2021-07-26T06:17:57.884 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:17:57.884 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:17:57.884 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:17:57.885 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:17:57.885 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:15:05.079001+0000", 2021-07-26T06:17:57.885 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:17:57.885 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.886 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.886 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:57.886 INFO:tasks.workunit.client.0.smithi094.stdout: "down_stamp": "2021-07-26T06:17:22.236809+0000", 2021-07-26T06:17:57.886 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_probability": 0, 2021-07-26T06:17:57.886 INFO:tasks.workunit.client.0.smithi094.stdout: "laggy_interval": 0, 2021-07-26T06:17:57.887 INFO:tasks.workunit.client.0.smithi094.stdout: "features": 4540138303579357183, 2021-07-26T06:17:57.887 INFO:tasks.workunit.client.0.smithi094.stdout: "old_weight": 0, 2021-07-26T06:17:57.887 INFO:tasks.workunit.client.0.smithi094.stdout: "last_purged_snaps_scrub": "2021-07-26T06:15:17.547758+0000", 2021-07-26T06:17:57.887 INFO:tasks.workunit.client.0.smithi094.stdout: "dead_epoch": 0 2021-07-26T06:17:57.887 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.888 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.888 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap": [], 2021-07-26T06:17:57.888 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_upmap_items": [], 2021-07-26T06:17:57.888 INFO:tasks.workunit.client.0.smithi094.stdout: "pg_temp": [], 2021-07-26T06:17:57.888 INFO:tasks.workunit.client.0.smithi094.stdout: "primary_temp": [], 2021-07-26T06:17:57.889 INFO:tasks.workunit.client.0.smithi094.stdout: "blocklist": {}, 2021-07-26T06:17:57.889 INFO:tasks.workunit.client.0.smithi094.stdout: "erasure_code_profiles": { 2021-07-26T06:17:57.889 INFO:tasks.workunit.client.0.smithi094.stdout: "default": { 2021-07-26T06:17:57.890 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:17:57.890 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "2", 2021-07-26T06:17:57.890 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:17:57.890 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van" 2021-07-26T06:17:57.891 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.891 INFO:tasks.workunit.client.0.smithi094.stdout: "myprofile": { 2021-07-26T06:17:57.891 INFO:tasks.workunit.client.0.smithi094.stdout: "crush-device-class": "", 2021-07-26T06:17:57.891 INFO:tasks.workunit.client.0.smithi094.stdout: "crush-failure-domain": "osd", 2021-07-26T06:17:57.891 INFO:tasks.workunit.client.0.smithi094.stdout: "crush-root": "default", 2021-07-26T06:17:57.892 INFO:tasks.workunit.client.0.smithi094.stdout: "jerasure-per-chunk-alignment": "false", 2021-07-26T06:17:57.892 INFO:tasks.workunit.client.0.smithi094.stdout: "k": "2", 2021-07-26T06:17:57.892 INFO:tasks.workunit.client.0.smithi094.stdout: "m": "1", 2021-07-26T06:17:57.892 INFO:tasks.workunit.client.0.smithi094.stdout: "plugin": "jerasure", 2021-07-26T06:17:57.892 INFO:tasks.workunit.client.0.smithi094.stdout: "technique": "reed_sol_van", 2021-07-26T06:17:57.893 INFO:tasks.workunit.client.0.smithi094.stdout: "w": "8" 2021-07-26T06:17:57.893 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.893 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.894 INFO:tasks.workunit.client.0.smithi094.stdout: "removed_snaps_queue": [], 2021-07-26T06:17:57.894 INFO:tasks.workunit.client.0.smithi094.stdout: "new_removed_snaps": [], 2021-07-26T06:17:57.894 INFO:tasks.workunit.client.0.smithi094.stdout: "new_purged_snaps": [], 2021-07-26T06:17:57.894 INFO:tasks.workunit.client.0.smithi094.stdout: "crush_node_flags": {}, 2021-07-26T06:17:57.895 INFO:tasks.workunit.client.0.smithi094.stdout: "device_class_flags": {}, 2021-07-26T06:17:57.895 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode": { 2021-07-26T06:17:57.895 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_enabled": false, 2021-07-26T06:17:57.895 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_bucket_count": 0, 2021-07-26T06:17:57.895 INFO:tasks.workunit.client.0.smithi094.stdout: "degraded_stretch_mode": 0, 2021-07-26T06:17:57.896 INFO:tasks.workunit.client.0.smithi094.stdout: "recovering_stretch_mode": 0, 2021-07-26T06:17:57.897 INFO:tasks.workunit.client.0.smithi094.stdout: "stretch_mode_bucket": 0 2021-07-26T06:17:57.897 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.897 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.898 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_metadata": [ 2021-07-26T06:17:57.898 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.899 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:57.899 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:17:57.900 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6804/68688,v1:127.0.0.1:6805/68688]", 2021-07-26T06:17:57.900 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:17:57.900 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:17:57.901 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:17:57.901 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:17:57.902 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:17:57.902 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:17:57.902 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:17:57.902 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:17:57.903 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:17:57.903 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-07-26T06:17:57.903 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:17:57.903 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:17:57.903 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:17:57.904 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:17:57.904 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:17:57.904 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:17:57.904 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:17:57.904 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:17:57.905 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:17:57.905 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:17:57.905 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:17:57.905 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:17:57.905 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:17:57.906 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:17:57.906 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:17:57.906 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6802/68688,v1:127.0.0.1:6803/68688]", 2021-07-26T06:17:57.906 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:17:57.907 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/68688,v1:127.0.0.1:6809/68688]", 2021-07-26T06:17:57.907 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/68688,v1:127.0.0.1:6807/68688]", 2021-07-26T06:17:57.907 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:17:57.907 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:17:57.907 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:17:57.908 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:17:57.908 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:17:57.908 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:17:57.908 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:17:57.908 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:17:57.909 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:17:57.909 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/0", 2021-07-26T06:17:57.909 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:17:57.909 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:17:57.909 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:17:57.910 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.910 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.910 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:57.911 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:17:57.911 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/60227,v1:127.0.0.1:6813/60227]", 2021-07-26T06:17:57.911 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:17:57.911 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:17:57.912 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:17:57.912 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:17:57.912 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:17:57.912 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:17:57.912 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:17:57.913 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:17:57.913 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:17:57.913 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-07-26T06:17:57.913 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:17:57.913 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:17:57.914 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:17:57.914 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:17:57.914 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:17:57.914 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:17:57.914 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:17:57.915 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:17:57.915 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:17:57.915 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:17:57.915 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:17:57.916 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:17:57.916 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:17:57.916 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:17:57.916 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:17:57.916 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/60227,v1:127.0.0.1:6811/60227]", 2021-07-26T06:17:57.917 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:17:57.917 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/60227,v1:127.0.0.1:6817/60227]", 2021-07-26T06:17:57.917 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/60227,v1:127.0.0.1:6815/60227]", 2021-07-26T06:17:57.917 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:17:57.917 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:17:57.918 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:17:57.919 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:17:57.919 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:17:57.919 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:17:57.919 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:17:57.920 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:17:57.920 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:17:57.920 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/1", 2021-07-26T06:17:57.920 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:17:57.921 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:17:57.921 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:17:57.921 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.922 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.922 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:57.922 INFO:tasks.workunit.client.0.smithi094.stdout: "arch": "x86_64", 2021-07-26T06:17:57.922 INFO:tasks.workunit.client.0.smithi094.stdout: "back_addr": "[v2:127.0.0.1:6812/69366,v1:127.0.0.1:6813/69366]", 2021-07-26T06:17:57.923 INFO:tasks.workunit.client.0.smithi094.stdout: "back_iface": "", 2021-07-26T06:17:57.923 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs": "1", 2021-07-26T06:17:57.923 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_db": "0", 2021-07-26T06:17:57.923 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_dedicated_wal": "0", 2021-07-26T06:17:57.923 INFO:tasks.workunit.client.0.smithi094.stdout: "bluefs_single_shared_device": "1", 2021-07-26T06:17:57.924 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:17:57.924 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:17:57.924 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:17:57.924 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:17:57.924 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-07-26T06:17:57.925 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:17:57.925 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:17:57.925 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:17:57.925 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:17:57.925 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_release": "quincy", 2021-07-26T06:17:57.926 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version": "ceph version 17.0.0-6358-g197e564f (197e564fdaee12596de1dc8f3253d1704ae66d5d) quincy (dev)", 2021-07-26T06:17:57.926 INFO:tasks.workunit.client.0.smithi094.stdout: "ceph_version_short": "17.0.0-6358-g197e564f", 2021-07-26T06:17:57.926 INFO:tasks.workunit.client.0.smithi094.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-07-26T06:17:57.926 INFO:tasks.workunit.client.0.smithi094.stdout: "default_device_class": "hdd", 2021-07-26T06:17:57.926 INFO:tasks.workunit.client.0.smithi094.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LJNP", 2021-07-26T06:17:57.927 INFO:tasks.workunit.client.0.smithi094.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-07-26T06:17:57.927 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": "sda", 2021-07-26T06:17:57.927 INFO:tasks.workunit.client.0.smithi094.stdout: "distro": "ubuntu", 2021-07-26T06:17:57.927 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_description": "Ubuntu 20.04.2 LTS", 2021-07-26T06:17:57.928 INFO:tasks.workunit.client.0.smithi094.stdout: "distro_version": "20.04", 2021-07-26T06:17:57.928 INFO:tasks.workunit.client.0.smithi094.stdout: "front_addr": "[v2:127.0.0.1:6810/69366,v1:127.0.0.1:6811/69366]", 2021-07-26T06:17:57.928 INFO:tasks.workunit.client.0.smithi094.stdout: "front_iface": "", 2021-07-26T06:17:57.928 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/69366,v1:127.0.0.1:6817/69366]", 2021-07-26T06:17:57.928 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/69366,v1:127.0.0.1:6815/69366]", 2021-07-26T06:17:57.929 INFO:tasks.workunit.client.0.smithi094.stdout: "hostname": "smithi094", 2021-07-26T06:17:57.929 INFO:tasks.workunit.client.0.smithi094.stdout: "journal_rotational": "1", 2021-07-26T06:17:57.929 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_description": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021", 2021-07-26T06:17:57.929 INFO:tasks.workunit.client.0.smithi094.stdout: "kernel_version": "5.4.0-80-generic", 2021-07-26T06:17:57.929 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_swap_kb": "2097148", 2021-07-26T06:17:57.930 INFO:tasks.workunit.client.0.smithi094.stdout: "mem_total_kb": "32771620", 2021-07-26T06:17:57.930 INFO:tasks.workunit.client.0.smithi094.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-07-26T06:17:57.930 INFO:tasks.workunit.client.0.smithi094.stdout: "objectstore_numa_unknown_devices": "sda", 2021-07-26T06:17:57.931 INFO:tasks.workunit.client.0.smithi094.stdout: "os": "Linux", 2021-07-26T06:17:57.932 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_data": "td/divergent-priors/2", 2021-07-26T06:17:57.932 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:17:57.932 INFO:tasks.workunit.client.0.smithi094.stdout: "osdspec_affinity": "", 2021-07-26T06:17:57.932 INFO:tasks.workunit.client.0.smithi094.stdout: "rotational": "1" 2021-07-26T06:17:57.932 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.933 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.933 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_clean_epochs": { 2021-07-26T06:17:57.933 INFO:tasks.workunit.client.0.smithi094.stdout: "min_last_epoch_clean": 0, 2021-07-26T06:17:57.933 INFO:tasks.workunit.client.0.smithi094.stdout: "last_epoch_clean": { 2021-07-26T06:17:57.933 INFO:tasks.workunit.client.0.smithi094.stdout: "per_pool": [] 2021-07-26T06:17:57.934 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.934 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_epochs": [ 2021-07-26T06:17:57.934 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.934 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:57.935 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 46 2021-07-26T06:17:57.935 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.935 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.935 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:57.935 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 10 2021-07-26T06:17:57.936 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.936 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.936 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:57.936 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 49 2021-07-26T06:17:57.936 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.937 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.937 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.937 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_first_committed": 1, 2021-07-26T06:17:57.937 INFO:tasks.workunit.client.0.smithi094.stdout: "osdmap_last_committed": 50, 2021-07-26T06:17:57.937 INFO:tasks.workunit.client.0.smithi094.stdout: "crushmap": { 2021-07-26T06:17:57.938 INFO:tasks.workunit.client.0.smithi094.stdout: "devices": [ 2021-07-26T06:17:57.938 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.938 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:57.938 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.0", 2021-07-26T06:17:57.938 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:17:57.939 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.939 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.939 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:57.939 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.1", 2021-07-26T06:17:57.939 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:17:57.940 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.942 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.942 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:57.943 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd.2", 2021-07-26T06:17:57.944 INFO:tasks.workunit.client.0.smithi094.stdout: "class": "hdd" 2021-07-26T06:17:57.944 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.944 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.944 INFO:tasks.workunit.client.0.smithi094.stdout: "types": [ 2021-07-26T06:17:57.945 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.947 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 0, 2021-07-26T06:17:57.947 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "osd" 2021-07-26T06:17:57.948 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.948 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.948 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:17:57.948 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "host" 2021-07-26T06:17:57.949 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.949 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.949 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 2, 2021-07-26T06:17:57.949 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "chassis" 2021-07-26T06:17:57.950 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.950 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.950 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 3, 2021-07-26T06:17:57.950 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "rack" 2021-07-26T06:17:57.951 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.951 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.952 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 4, 2021-07-26T06:17:57.953 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "row" 2021-07-26T06:17:57.953 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.953 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.954 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 5, 2021-07-26T06:17:57.954 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pdu" 2021-07-26T06:17:57.954 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.954 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.955 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 6, 2021-07-26T06:17:57.955 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "pod" 2021-07-26T06:17:57.956 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.956 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.956 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 7, 2021-07-26T06:17:57.956 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "room" 2021-07-26T06:17:57.957 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.958 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.958 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 8, 2021-07-26T06:17:57.958 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "datacenter" 2021-07-26T06:17:57.958 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.958 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.959 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 9, 2021-07-26T06:17:57.959 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "zone" 2021-07-26T06:17:57.959 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.959 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.960 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 10, 2021-07-26T06:17:57.960 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "region" 2021-07-26T06:17:57.960 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.960 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.961 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:17:57.961 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "root" 2021-07-26T06:17:57.961 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.961 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.962 INFO:tasks.workunit.client.0.smithi094.stdout: "buckets": [ 2021-07-26T06:17:57.962 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.963 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -1, 2021-07-26T06:17:57.963 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default", 2021-07-26T06:17:57.963 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:17:57.963 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:17:57.964 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:57.964 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:57.964 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:57.964 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:57.965 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.965 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:17:57.965 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:57.966 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:57.966 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.966 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.966 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.967 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.967 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -2, 2021-07-26T06:17:57.968 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "default~hdd", 2021-07-26T06:17:57.968 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 11, 2021-07-26T06:17:57.968 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "root", 2021-07-26T06:17:57.968 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:57.969 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:57.969 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:57.969 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:57.969 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.969 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:17:57.970 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:57.970 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:57.970 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.970 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.971 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.971 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.971 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -3, 2021-07-26T06:17:57.971 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094", 2021-07-26T06:17:57.971 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:17:57.972 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:17:57.972 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:57.972 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:57.972 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:57.973 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:57.973 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.973 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:57.973 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:57.973 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:57.974 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.974 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.974 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:57.974 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:57.975 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:17:57.975 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.975 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.975 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:57.975 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:57.976 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:17:57.976 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.977 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.977 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.977 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.977 INFO:tasks.workunit.client.0.smithi094.stdout: "id": -4, 2021-07-26T06:17:57.977 INFO:tasks.workunit.client.0.smithi094.stdout: "name": "smithi094~hdd", 2021-07-26T06:17:57.978 INFO:tasks.workunit.client.0.smithi094.stdout: "type_id": 1, 2021-07-26T06:17:57.978 INFO:tasks.workunit.client.0.smithi094.stdout: "type_name": "host", 2021-07-26T06:17:57.979 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 19206, 2021-07-26T06:17:57.979 INFO:tasks.workunit.client.0.smithi094.stdout: "alg": "straw2", 2021-07-26T06:17:57.979 INFO:tasks.workunit.client.0.smithi094.stdout: "hash": "rjenkins1", 2021-07-26T06:17:57.979 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:17:57.979 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.980 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 0, 2021-07-26T06:17:57.980 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:57.980 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 0 2021-07-26T06:17:57.981 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.981 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.981 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 1, 2021-07-26T06:17:57.981 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:57.981 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 1 2021-07-26T06:17:57.982 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.982 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.982 INFO:tasks.workunit.client.0.smithi094.stdout: "id": 2, 2021-07-26T06:17:57.982 INFO:tasks.workunit.client.0.smithi094.stdout: "weight": 6402, 2021-07-26T06:17:57.982 INFO:tasks.workunit.client.0.smithi094.stdout: "pos": 2 2021-07-26T06:17:57.983 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.983 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.983 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.983 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:57.984 INFO:tasks.workunit.client.0.smithi094.stdout: "rules": [ 2021-07-26T06:17:57.984 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.984 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 0, 2021-07-26T06:17:57.984 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "replicated_rule", 2021-07-26T06:17:57.984 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 1, 2021-07-26T06:17:57.985 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:17:57.985 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.986 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:17:57.986 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:17:57.986 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:17:57.987 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.987 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.987 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_firstn", 2021-07-26T06:17:57.988 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:17:57.988 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:17:57.990 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.990 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.990 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:17:57.990 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:57.991 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:57.991 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.991 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.991 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_id": 1, 2021-07-26T06:17:57.991 INFO:tasks.workunit.client.0.smithi094.stdout: "rule_name": "test", 2021-07-26T06:17:57.992 INFO:tasks.workunit.client.0.smithi094.stdout: "type": 3, 2021-07-26T06:17:57.992 INFO:tasks.workunit.client.0.smithi094.stdout: "steps": [ 2021-07-26T06:17:57.992 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.992 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "set_chooseleaf_tries", 2021-07-26T06:17:57.993 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 5 2021-07-26T06:17:57.993 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.993 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.993 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "set_choose_tries", 2021-07-26T06:17:57.996 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 100 2021-07-26T06:17:57.997 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.997 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.998 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "take", 2021-07-26T06:17:57.998 INFO:tasks.workunit.client.0.smithi094.stdout: "item": -1, 2021-07-26T06:17:57.998 INFO:tasks.workunit.client.0.smithi094.stdout: "item_name": "default" 2021-07-26T06:17:57.998 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:57.999 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:57.999 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "choose_indep", 2021-07-26T06:17:57.999 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 0, 2021-07-26T06:17:57.999 INFO:tasks.workunit.client.0.smithi094.stdout: "type": "osd" 2021-07-26T06:17:58.000 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.000 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.000 INFO:tasks.workunit.client.0.smithi094.stdout: "op": "emit" 2021-07-26T06:17:58.001 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.001 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:17:58.001 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.001 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:58.001 INFO:tasks.workunit.client.0.smithi094.stdout: "tunables": { 2021-07-26T06:17:58.002 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_tries": 0, 2021-07-26T06:17:58.002 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_local_fallback_tries": 0, 2021-07-26T06:17:58.002 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_total_tries": 50, 2021-07-26T06:17:58.002 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_descend_once": 1, 2021-07-26T06:17:58.003 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_vary_r": 1, 2021-07-26T06:17:58.003 INFO:tasks.workunit.client.0.smithi094.stdout: "chooseleaf_stable": 1, 2021-07-26T06:17:58.003 INFO:tasks.workunit.client.0.smithi094.stdout: "straw_calc_version": 1, 2021-07-26T06:17:58.003 INFO:tasks.workunit.client.0.smithi094.stdout: "allowed_bucket_algs": 54, 2021-07-26T06:17:58.003 INFO:tasks.workunit.client.0.smithi094.stdout: "profile": "jewel", 2021-07-26T06:17:58.004 INFO:tasks.workunit.client.0.smithi094.stdout: "optimal_tunables": 1, 2021-07-26T06:17:58.004 INFO:tasks.workunit.client.0.smithi094.stdout: "legacy_tunables": 0, 2021-07-26T06:17:58.004 INFO:tasks.workunit.client.0.smithi094.stdout: "minimum_required_version": "jewel", 2021-07-26T06:17:58.004 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables": 1, 2021-07-26T06:17:58.004 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables2": 1, 2021-07-26T06:17:58.005 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v2_rules": 1, 2021-07-26T06:17:58.005 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables3": 1, 2021-07-26T06:17:58.005 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v3_rules": 0, 2021-07-26T06:17:58.005 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v4_buckets": 1, 2021-07-26T06:17:58.006 INFO:tasks.workunit.client.0.smithi094.stdout: "require_feature_tunables5": 1, 2021-07-26T06:17:58.006 INFO:tasks.workunit.client.0.smithi094.stdout: "has_v5_rules": 0 2021-07-26T06:17:58.006 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.006 INFO:tasks.workunit.client.0.smithi094.stdout: "choose_args": {} 2021-07-26T06:17:58.006 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.007 INFO:tasks.workunit.client.0.smithi094.stdout: "fsmap": { 2021-07-26T06:17:58.007 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 1, 2021-07-26T06:17:58.007 INFO:tasks.workunit.client.0.smithi094.stdout: "default_fscid": -1, 2021-07-26T06:17:58.010 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": { 2021-07-26T06:17:58.010 INFO:tasks.workunit.client.0.smithi094.stdout: "compat": {}, 2021-07-26T06:17:58.010 INFO:tasks.workunit.client.0.smithi094.stdout: "ro_compat": {}, 2021-07-26T06:17:58.011 INFO:tasks.workunit.client.0.smithi094.stdout: "incompat": { 2021-07-26T06:17:58.011 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_1": "base v0.20", 2021-07-26T06:17:58.012 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_2": "client writeable ranges", 2021-07-26T06:17:58.012 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_3": "default file layouts on dirs", 2021-07-26T06:17:58.012 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_4": "dir inode in separate object", 2021-07-26T06:17:58.013 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_5": "mds uses versioned encoding", 2021-07-26T06:17:58.013 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_6": "dirfrag is stored in omap", 2021-07-26T06:17:58.013 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_8": "no anchor table", 2021-07-26T06:17:58.013 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_9": "file layout v2", 2021-07-26T06:17:58.014 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_10": "snaprealm v2" 2021-07-26T06:17:58.014 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.015 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.015 INFO:tasks.workunit.client.0.smithi094.stdout: "feature_flags": { 2021-07-26T06:17:58.016 INFO:tasks.workunit.client.0.smithi094.stdout: "enable_multiple": true, 2021-07-26T06:17:58.016 INFO:tasks.workunit.client.0.smithi094.stdout: "ever_enabled_multiple": true 2021-07-26T06:17:58.017 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.017 INFO:tasks.workunit.client.0.smithi094.stdout: "standbys": [], 2021-07-26T06:17:58.017 INFO:tasks.workunit.client.0.smithi094.stdout: "filesystems": [] 2021-07-26T06:17:58.018 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.018 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_first_committed": 1, 2021-07-26T06:17:58.018 INFO:tasks.workunit.client.0.smithi094.stdout: "mdsmap_last_committed": 1, 2021-07-26T06:17:58.019 INFO:tasks.workunit.client.0.smithi094.stdout: "auth": { 2021-07-26T06:17:58.019 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:17:58.019 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 5, 2021-07-26T06:17:58.019 INFO:tasks.workunit.client.0.smithi094.stdout: "num_secrets": 3 2021-07-26T06:17:58.019 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.020 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:17:58.020 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_active": 1, 2021-07-26T06:17:58.021 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_unknown": 0, 2021-07-26T06:17:58.021 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osd": 3, 2021-07-26T06:17:58.021 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_sum": { 2021-07-26T06:17:58.021 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:17:58.022 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 62460, 2021-07-26T06:17:58.022 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 101, 2021-07-26T06:17:58.022 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 1, 2021-07-26T06:17:58.022 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 303, 2021-07-26T06:17:58.024 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:17:58.025 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:17:58.025 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 101, 2021-07-26T06:17:58.025 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:17:58.027 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:17:58.027 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 101, 2021-07-26T06:17:58.028 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:17:58.028 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:17:58.028 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:17:58.029 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 101, 2021-07-26T06:17:58.029 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 101, 2021-07-26T06:17:58.030 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:17:58.030 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:17:58.031 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:17:58.031 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:17:58.032 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:17:58.032 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:17:58.033 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:17:58.033 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:17:58.034 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:17:58.034 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:17:58.035 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:17:58.035 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:17:58.035 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:17:58.036 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:17:58.036 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:17:58.037 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:17:58.037 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:17:58.037 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:17:58.038 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:17:58.038 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:17:58.039 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:17:58.039 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:17:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:17:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:17:58.040 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:17:58.041 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.041 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:17:58.042 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:17:58.042 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:17:58.043 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:58.043 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 0, 2021-07-26T06:17:58.044 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 0, 2021-07-26T06:17:58.044 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:58.045 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:58.045 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:58.046 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:17:58.046 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:17:58.046 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.047 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 18, 2021-07-26T06:17:58.047 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 18, 2021-07-26T06:17:58.048 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 3, 2021-07-26T06:17:58.048 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 3, 2021-07-26T06:17:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 0 2021-07-26T06:17:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.049 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum": { 2021-07-26T06:17:58.050 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:17:58.050 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:17:58.051 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:17:58.052 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:17:58.052 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:17:58.053 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:17:58.053 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:17:58.054 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 19332, 2021-07-26T06:17:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2276, 2021-07-26T06:17:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 4, 2021-07-26T06:17:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 17019, 2021-07-26T06:17:58.055 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314553468, 2021-07-26T06:17:58.056 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:17:58.056 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:17:58.056 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322102751232, 2021-07-26T06:17:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2330624, 2021-07-26T06:17:58.057 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 1592509, 2021-07-26T06:17:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 4866, 2021-07-26T06:17:58.058 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 17427710 2021-07-26T06:17:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:17:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:17:58.059 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:17:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:17:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:17:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:17:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:17:58.060 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:17:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:17:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:17:58.061 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:17:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:17:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:17:58.062 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:17:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_sum_by_class": { 2021-07-26T06:17:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "hdd": { 2021-07-26T06:17:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "up_from": 0, 2021-07-26T06:17:58.063 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 0, 2021-07-26T06:17:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pgs": 3, 2021-07-26T06:17:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "num_osds": 3, 2021-07-26T06:17:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_osds": 3, 2021-07-26T06:17:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "num_per_pool_omap_osds": 3, 2021-07-26T06:17:58.064 INFO:tasks.workunit.client.0.smithi094.stdout: "kb": 314572800, 2021-07-26T06:17:58.065 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used": 19332, 2021-07-26T06:17:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_data": 2276, 2021-07-26T06:17:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_omap": 4, 2021-07-26T06:17:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_used_meta": 17019, 2021-07-26T06:17:58.066 INFO:tasks.workunit.client.0.smithi094.stdout: "kb_avail": 314553468, 2021-07-26T06:17:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "statfs": { 2021-07-26T06:17:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 322122547200, 2021-07-26T06:17:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 322102751232, 2021-07-26T06:17:58.067 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 2330624, 2021-07-26T06:17:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 1592509, 2021-07-26T06:17:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:58.068 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 4866, 2021-07-26T06:17:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 17427710 2021-07-26T06:17:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "hb_peers": [], 2021-07-26T06:17:58.069 INFO:tasks.workunit.client.0.smithi094.stdout: "snap_trim_queue_len": 0, 2021-07-26T06:17:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "num_snap_trimming": 0, 2021-07-26T06:17:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shards_repaired": 0, 2021-07-26T06:17:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "op_queue_age_hist": { 2021-07-26T06:17:58.070 INFO:tasks.workunit.client.0.smithi094.stdout: "histogram": [], 2021-07-26T06:17:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: "upper_bound": 1 2021-07-26T06:17:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: "perf_stat": { 2021-07-26T06:17:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ms": 0, 2021-07-26T06:17:58.071 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ms": 0, 2021-07-26T06:17:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "commit_latency_ns": 0, 2021-07-26T06:17:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "apply_latency_ns": 0 2021-07-26T06:17:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.072 INFO:tasks.workunit.client.0.smithi094.stdout: "alerts": [], 2021-07-26T06:17:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: "network_ping_times": [] 2021-07-26T06:17:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: "pool_stats": [ 2021-07-26T06:17:58.073 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "poolid": 1, 2021-07-26T06:17:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg": 1, 2021-07-26T06:17:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "stat_sum": { 2021-07-26T06:17:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes": 62460, 2021-07-26T06:17:58.074 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects": 101, 2021-07-26T06:17:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_clones": 1, 2021-07-26T06:17:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "num_object_copies": 303, 2021-07-26T06:17:58.075 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing_on_primary": 0, 2021-07-26T06:17:58.077 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_missing": 0, 2021-07-26T06:17:58.077 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_degraded": 101, 2021-07-26T06:17:58.078 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_misplaced": 0, 2021-07-26T06:17:58.079 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_unfound": 0, 2021-07-26T06:17:58.079 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_dirty": 101, 2021-07-26T06:17:58.080 INFO:tasks.workunit.client.0.smithi094.stdout: "num_whiteouts": 0, 2021-07-26T06:17:58.081 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read": 0, 2021-07-26T06:17:58.081 INFO:tasks.workunit.client.0.smithi094.stdout: "num_read_kb": 0, 2021-07-26T06:17:58.082 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write": 101, 2021-07-26T06:17:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: "num_write_kb": 101, 2021-07-26T06:17:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: "num_scrub_errors": 0, 2021-07-26T06:17:58.083 INFO:tasks.workunit.client.0.smithi094.stdout: "num_shallow_scrub_errors": 0, 2021-07-26T06:17:58.085 INFO:tasks.workunit.client.0.smithi094.stdout: "num_deep_scrub_errors": 0, 2021-07-26T06:17:58.085 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_recovered": 0, 2021-07-26T06:17:58.086 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_recovered": 0, 2021-07-26T06:17:58.086 INFO:tasks.workunit.client.0.smithi094.stdout: "num_keys_recovered": 0, 2021-07-26T06:17:58.087 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_omap": 0, 2021-07-26T06:17:58.087 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_hit_set_archive": 0, 2021-07-26T06:17:58.088 INFO:tasks.workunit.client.0.smithi094.stdout: "num_bytes_hit_set_archive": 0, 2021-07-26T06:17:58.088 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush": 0, 2021-07-26T06:17:58.088 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_kb": 0, 2021-07-26T06:17:58.089 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict": 0, 2021-07-26T06:17:58.089 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_kb": 0, 2021-07-26T06:17:58.090 INFO:tasks.workunit.client.0.smithi094.stdout: "num_promote": 0, 2021-07-26T06:17:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_high": 0, 2021-07-26T06:17:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: "num_flush_mode_low": 0, 2021-07-26T06:17:58.091 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_some": 0, 2021-07-26T06:17:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "num_evict_mode_full": 0, 2021-07-26T06:17:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_pinned": 0, 2021-07-26T06:17:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "num_legacy_snapsets": 0, 2021-07-26T06:17:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "num_large_omap_objects": 0, 2021-07-26T06:17:58.092 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_manifest": 0, 2021-07-26T06:17:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_bytes": 0, 2021-07-26T06:17:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: "num_omap_keys": 0, 2021-07-26T06:17:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: "num_objects_repaired": 0 2021-07-26T06:17:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.093 INFO:tasks.workunit.client.0.smithi094.stdout: "store_stats": { 2021-07-26T06:17:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: "total": 0, 2021-07-26T06:17:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: "available": 0, 2021-07-26T06:17:58.094 INFO:tasks.workunit.client.0.smithi094.stdout: "internally_reserved": 0, 2021-07-26T06:17:58.095 INFO:tasks.workunit.client.0.smithi094.stdout: "allocated": 1241088, 2021-07-26T06:17:58.096 INFO:tasks.workunit.client.0.smithi094.stdout: "data_stored": 1241088, 2021-07-26T06:17:58.096 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed": 0, 2021-07-26T06:17:58.097 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_allocated": 0, 2021-07-26T06:17:58.097 INFO:tasks.workunit.client.0.smithi094.stdout: "data_compressed_original": 0, 2021-07-26T06:17:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "omap_allocated": 0, 2021-07-26T06:17:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "internal_metadata": 0 2021-07-26T06:17:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.098 INFO:tasks.workunit.client.0.smithi094.stdout: "log_size": 18, 2021-07-26T06:17:58.099 INFO:tasks.workunit.client.0.smithi094.stdout: "ondisk_log_size": 18, 2021-07-26T06:17:58.099 INFO:tasks.workunit.client.0.smithi094.stdout: "up": 3, 2021-07-26T06:17:58.100 INFO:tasks.workunit.client.0.smithi094.stdout: "acting": 3, 2021-07-26T06:17:58.100 INFO:tasks.workunit.client.0.smithi094.stdout: "num_store_stats": 3 2021-07-26T06:17:58.101 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.101 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:58.102 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_stats": [ 2021-07-26T06:17:58.102 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 193273528330 2021-07-26T06:17:58.104 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.104 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.105 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:58.106 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 42949672983 2021-07-26T06:17:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.107 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:58.108 INFO:tasks.workunit.client.0.smithi094.stdout: "seq": 210453397513 2021-07-26T06:17:58.108 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.109 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:58.109 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_state": [ 2021-07-26T06:17:58.110 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.110 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "active+undersized+degraded", 2021-07-26T06:17:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: "num": 1 2021-07-26T06:17:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.111 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:58.112 INFO:tasks.workunit.client.0.smithi094.stdout: "num_pg_by_osd": [ 2021-07-26T06:17:58.112 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.113 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2147483647, 2021-07-26T06:17:58.113 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:17:58.113 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:17:58.114 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.115 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 2, 2021-07-26T06:17:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:17:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:17:58.117 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:58.118 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.118 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.119 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 0, 2021-07-26T06:17:58.119 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 1, 2021-07-26T06:17:58.120 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 1, 2021-07-26T06:17:58.120 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:58.121 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.121 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.122 INFO:tasks.workunit.client.0.smithi094.stdout: "osd": 1, 2021-07-26T06:17:58.122 INFO:tasks.workunit.client.0.smithi094.stdout: "num_primary_pg": 0, 2021-07-26T06:17:58.123 INFO:tasks.workunit.client.0.smithi094.stdout: "num_acting_pg": 0, 2021-07-26T06:17:58.123 INFO:tasks.workunit.client.0.smithi094.stdout: "num_up_not_acting_pg": 0 2021-07-26T06:17:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:58.124 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": [ 2021-07-26T06:17:58.125 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:17:58.125 INFO:tasks.workunit.client.0.smithi094.stdout: "pool": 1, 2021-07-26T06:17:58.126 INFO:tasks.workunit.client.0.smithi094.stdout: "purged_snaps": {} 2021-07-26T06:17:58.126 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:17:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: "servicemap": { 2021-07-26T06:17:58.127 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": 5, 2021-07-26T06:17:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: "modified": "2021-07-26T06:16:18.814309+0000", 2021-07-26T06:17:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: "services": {} 2021-07-26T06:17:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:17:58.128 INFO:tasks.workunit.client.0.smithi094.stdout: "paxos": { 2021-07-26T06:17:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: "first_committed": 1, 2021-07-26T06:17:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: "last_committed": 543, 2021-07-26T06:17:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: "last_pn": 0, 2021-07-26T06:17:58.129 INFO:tasks.workunit.client.0.smithi094.stdout: "accepted_pn": 0 2021-07-26T06:17:58.130 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:17:58.130 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:17:58.131 INFO:tasks.workunit.client.0.smithi094.stderr:report 1367861140 2021-07-26T06:17:58.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: return 1 2021-07-26T06:17:58.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2021-07-26T06:17:58.132 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery 2021-07-26T06:17:58.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:17:58.135 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:17:58.135 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 101 0 0 62460 0 0 18 18 active+undersized+degraded 2021-07-26T06:17:23.293334+0000 42'102 50:134 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:17:58.135 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:17:58.136 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:17:58.137 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:17:58.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:17:58.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2021-07-26T06:17:58.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:17:58.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:17:58.139 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:17:58.139 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:17:58.139 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:17:58.139 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.0.asok 2021-07-26T06:17:58.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-07-26T06:17:58.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.0.asok set_recovery_delay 100000 2021-07-26T06:17:58.190 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:17:58.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-07-26T06:17:58.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2021-07-26T06:17:58.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-07-26T06:17:58.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-07-26T06:17:58.199 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:17:58.199 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:17:58.199 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:17:58.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.2.asok 2021-07-26T06:17:58.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-07-26T06:17:58.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.2.asok set_recovery_delay 100000 2021-07-26T06:17:58.286 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:17:58.287 INFO:tasks.workunit.client.0.smithi094.stdout:revive divergent 1 2021-07-26T06:17:58.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2021-07-26T06:17:58.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:17:58.603 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:17:58.603 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 101 0 0 62460 0 0 18 18 active+undersized+degraded 2021-07-26T06:17:23.293334+0000 42'102 50:134 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:17:58.604 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:17:58.604 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:17:58.604 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:17:58.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2021-07-26T06:17:59.493 INFO:tasks.workunit.client.0.smithi094.stderr:noup is set 2021-07-26T06:17:59.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-07-26T06:17:59.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:17:59.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:17:59.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:17:59.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:17:59.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:17:59.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:17:59.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:17:59.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:17:59.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:17:59.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:17:59.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:17:59.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:17:59.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:17:59.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:17:59.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:17:59.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:17:59.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:17:59.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:17:59.515 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:17:59.516 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:17:59.516 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:17:59.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:17:59.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:17:59.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:17:59.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:17:59.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:17:59.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:17:59.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:17:59.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:17:59.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:17:59.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:17:59.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:17:59.524 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:17:59.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:17:59.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:17:59.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:17:59.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:17:59.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:17:59.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:17:59.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:17:59.549 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:59.535+0000 7f5f0412cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:59.553 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:59.539+0000 7f5f0412cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:59.554 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:17:59.543+0000 7f5f0412cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:17:59.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: activate_osd: return 0 2021-07-26T06:17:59.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2021-07-26T06:18:00.103 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:00.091+0000 7f5f0412cf40 -1 Falling back to public interface 2021-07-26T06:18:02.730 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:02.719+0000 7f5f0412cf40 -1 osd.1 30 log_to_monitors {default=true} 2021-07-26T06:18:04.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2021-07-26T06:18:04.884 INFO:tasks.workunit.client.0.smithi094.stdout:delay recovery divergent 2021-07-26T06:18:04.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:18:05.229 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:18:05.229 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 101 0 0 62460 0 0 18 18 active+undersized+degraded 2021-07-26T06:17:23.293334+0000 42'102 52:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:18:05.229 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:18:05.230 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:18:05.232 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:18:05.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2021-07-26T06:18:05.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:18:05.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:18:05.242 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:18:05.243 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:05.243 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:18:05.243 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.18330/ceph-osd.1.asok 2021-07-26T06:18:05.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2021-07-26T06:18:05.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.18330/ceph-osd.1.asok set_recovery_delay 100000 2021-07-26T06:18:05.328 INFO:tasks.workunit.client.0.smithi094.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-07-26T06:18:05.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2021-07-26T06:18:06.006 INFO:tasks.workunit.client.0.smithi094.stderr:noup is unset 2021-07-26T06:18:06.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2021-07-26T06:18:06.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:18:06.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:18:06.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:18:06.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:18:06.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:06.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:18:06.019 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:18:06.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:06.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:18:06.545 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 45 up_thru 49 down_at 44 last_clean_interval [37,42) [v2:127.0.0.1:6802/68688,v1:127.0.0.1:6803/68688] [v2:127.0.0.1:6804/68688,v1:127.0.0.1:6805/68688] exists,up a83a0bc5-6fa4-4572-8fce-7878e9f50f58 2021-07-26T06:18:06.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:18:06.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:18:06.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:18:06.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2021-07-26T06:18:06.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:18:06.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:18:06.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:18:06.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:18:06.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:06.550 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:18:06.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:18:06.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:06.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:18:07.094 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/71521,v1:127.0.0.1:6819/71521] [v2:127.0.0.1:6820/71521,v1:127.0.0.1:6821/71521] exists,up ffede5c8-cf1c-400b-9ab1-242a38919ce0 2021-07-26T06:18:07.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:18:07.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:18:07.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:18:07.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2021-07-26T06:18:07.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:18:07.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:18:07.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:18:07.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:18:07.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:07.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:18:07.098 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:18:07.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:07.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:18:07.477 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 49 up_thru 0 down_at 48 last_clean_interval [39,42) [v2:127.0.0.1:6810/69366,v1:127.0.0.1:6811/69366] [v2:127.0.0.1:6812/69366,v1:127.0.0.1:6813/69366] exists,up 93b45a7f-022d-45f0-a0fa-7fb07deeeabd 2021-07-26T06:18:07.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:18:07.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:18:07.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:18:07.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:18:07.853 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:18:07.853 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 101 0 0 62460 0 0 18 18 active+undersized+degraded 2021-07-26T06:17:23.293334+0000 42'102 52:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:18:07.854 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:18:07.854 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:18:07.856 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:18:07.864 INFO:tasks.workunit.client.0.smithi094.stdout:wait for peering 2021-07-26T06:18:07.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2021-07-26T06:18:07.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:18:08.194 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:18:08.194 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 0 101 0 0 62460 0 0 18 18 active+undersized+degraded 2021-07-26T06:17:23.293334+0000 42'102 52:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:18:08.195 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:18:08.195 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:18:08.197 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:18:08.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2021-07-26T06:18:19.407 INFO:tasks.workunit.client.0.smithi094.stdout:killing divergent 1 2021-07-26T06:18:19.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2021-07-26T06:18:19.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:18:19.761 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:18:19.761 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 4 4 0 0 62460 0 0 12 12 active+recovering+degraded+wait 2021-07-26T06:18:06.770753+0000 42'102 56:127 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:18:19.762 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:18:19.762 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:18:19.763 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:18:19.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2021-07-26T06:18:19.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:18:19.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:18:19.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:18:19.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:18:19.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:18:19.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:18:19.885 INFO:tasks.workunit.client.0.smithi094.stdout:reviving divergent 1 2021-07-26T06:18:19.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2021-07-26T06:18:19.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:18:20.213 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:18:20.214 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 101 4 4 0 0 62460 0 0 12 12 active+recovering+degraded+wait 2021-07-26T06:18:06.770753+0000 42'102 56:127 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:18:20.215 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:18:20.215 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:18:20.221 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:18:20.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-07-26T06:18:20.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/divergent-priors 2021-07-26T06:18:20.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:18:20.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:18:20.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:18:20.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/divergent-priors/1 2021-07-26T06:18:20.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-07-26T06:18:20.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:18:20.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:18:20.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:18:20.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-07-26T06:18:20.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-07-26T06:18:20.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:18:20.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:18:20.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:18:20.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-07-26T06:18:20.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:18:20.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:18:20.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:18:20.246 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:18:20.246 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:20.247 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:18:20.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:18:20.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' 2021-07-26T06:18:20.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:18:20.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-07-26T06:18:20.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-07-26T06:18:20.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:18:20.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:18:20.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:18:20.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:18:20.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:18:20.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/divergent-priors/1 2021-07-26T06:18:20.250 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:18:20.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:18:20.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=01dff7b3-ca67-411f-8a1d-dda47809b70a --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.18330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:18:20.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/divergent-priors/1/whoami 2021-07-26T06:18:20.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:18:20.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:18:20.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:18:20.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:18:20.267 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:20.250+0000 7f3b76fd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:20.267 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:20.250+0000 7f3b76fd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:20.267 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:20.254+0000 7f3b76fd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:20.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:18:20.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:18:20.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:18:20.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:18:20.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:18:20.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:20.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:18:20.614 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:18:20.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:20.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:18:20.808 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:20.794+0000 7f3b76fd0f40 -1 Falling back to public interface 2021-07-26T06:18:20.972 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/71521,v1:127.0.0.1:6819/71521] [v2:127.0.0.1:6820/71521,v1:127.0.0.1:6821/71521] exists,up ffede5c8-cf1c-400b-9ab1-242a38919ce0 2021-07-26T06:18:20.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:18:20.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:18:20.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:18:20.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2021-07-26T06:18:23.474 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:23.462+0000 7f3b76fd0f40 -1 osd.1 56 log_to_monitors {default=true} 2021-07-26T06:18:40.976 INFO:tasks.workunit.client.0.smithi094.stdout:allowing recovery 2021-07-26T06:18:40.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2021-07-26T06:18:40.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:18:41.389 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:18:41.396 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 102 0 0 0 0 63077 0 0 13 13 active+clean 2021-07-26T06:18:25.724052+0000 56'103 60:151 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:18:41.397 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:18:41.397 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:18:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:18:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-07-26T06:18:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2021-07-26T06:18:41.543 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:18:41.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-07-26T06:18:41.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2021-07-26T06:18:41.702 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:18:41.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-07-26T06:18:41.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2021-07-26T06:18:41.845 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T06:18:41.854 INFO:tasks.workunit.client.0.smithi094.stdout:reading divergent objects 2021-07-26T06:18:41.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2021-07-26T06:18:41.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2021-07-26T06:18:42.207 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:18:42.207 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 102 0 0 0 0 63077 0 0 13 13 active+clean 2021-07-26T06:18:25.724052+0000 56'103 60:151 [1,0,2] 1 [1,0,2] 1 0'0 2021-07-26T06:15:22.041775+0000 0'0 2021-07-26T06:15:22.041775+0000 0 2021-07-26T06:18:42.208 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:18:42.208 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:18:42.209 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:18:42.219 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2021-07-26T06:18:42.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2021-07-26T06:18:42.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:18:42.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2021-07-26T06:18:42.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:18:42.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2021-07-26T06:18:42.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:18:42.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2021-07-26T06:18:42.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-07-26T06:18:42.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2021-07-26T06:18:42.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2021-07-26T06:18:42.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-07-26T06:18:42.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:18:42.429 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-07-26T06:16:01.479191+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.429 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-07-26T06:16:01.479191+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.430 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:18:42.430 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2021-07-26T06:18:42.431 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2021-07-26T06:18:42.433 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2021-07-26T06:18:42.433 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4569.0:1 2021-07-26T06:16:09.226865+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.433 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4569.0:1 2021-07-26T06:16:09.226865+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.433 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:18:42.434 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2021-07-26T06:18:42.434 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2021-07-26T06:18:42.437 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:18:42.437 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2021-07-26T06:18:42.437 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4569.0:1 2021-07-26T06:16:09.226865+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.437 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2021-07-26T06:18:42.438 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4560.0:1 2021-07-26T06:16:07.908229+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4566.0:1 2021-07-26T06:16:08.913536+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.440 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4560.0:1 2021-07-26T06:16:07.908229+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.440 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4566.0:1 2021-07-26T06:16:08.913536+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.440 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-07-26T06:18:42.441 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2021-07-26T06:18:42.441 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-07-26T06:18:42.441 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-07-26T06:18:42.442 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2021-07-26T06:18:42.442 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4566.0:1 2021-07-26T06:16:08.913536+0000 0 ObjectCleanRegions clean_offsets: [760~18446744073709550855], clean_omap: 1, new_object: 0 2021-07-26T06:18:42.442 INFO:tasks.workunit.client.0.smithi094.stdout:td/divergent-priors/osd.1.log:2021-07-26T06:18:06.603+0000 7f5eea646700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-07-26T06:18:42.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-07-26T06:18:42.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log td/divergent-priors/osd.0.log 2021-07-26T06:18:42.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2021-07-26T06:18:42.444 INFO:tasks.workunit.client.0.smithi094.stdout:success 2021-07-26T06:18:42.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2021-07-26T06:18:42.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T06:18:42.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T06:18:42.951 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T06:18:42.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2021-07-26T06:18:42.966 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:18:42.966 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:18:42.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:18:42.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:18:42.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:18:44.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:18:44.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-07-26T06:18:44.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:18:44.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:18:44.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:18:44.287 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:18:44.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:18:44.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:18:44.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:18:44.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:18:44.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:18:44.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:18:44.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:18:44.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:18:44.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:18:44.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:18:44.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:18:44.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:18:44.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:18:44.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:18:44.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:18:44.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:18:44.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:18:44.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:18:44.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:18:44.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:44.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:18:44.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:18:44.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:18:44.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:18:44.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:18:44.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/divergent-priors 0 2021-07-26T06:18:44.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-07-26T06:18:44.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:18:44.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-07-26T06:18:44.335 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:18:44.335 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:18:44.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:18:44.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:18:44.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:18:44.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:18:44.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:18:44.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:18:44.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:18:44.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:18:44.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:18:44.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:18:44.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:18:44.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:18:44.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:18:44.352 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:18:44.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:18:44.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:18:44.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-07-26T06:18:44.352 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:18:44.353 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:44.353 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.18330 2021-07-26T06:18:44.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.18330 2021-07-26T06:18:44.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:18:44.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:18:44.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:18:44.355 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:18:44.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:18:44.375 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2021-07-26T06:18:44.376 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2021-07-26T06:18:44.454 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:18:44.455 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:18:44.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:18:44.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:18:44.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:18:44.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:18:44.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:18:44.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:18:44.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:18:44.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:18:44.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/ec-error-rollforward 2021-07-26T06:18:44.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2021-07-26T06:18:44.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2021-07-26T06:18:44.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2021-07-26T06:18:44.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2021-07-26T06:18:44.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2021-07-26T06:18:44.462 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2021-07-26T06:18:44.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none ' 2021-07-26T06:18:44.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2021-07-26T06:18:44.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2021-07-26T06:18:44.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2021-07-26T06:18:44.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2021-07-26T06:18:44.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2021-07-26T06:18:44.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2021-07-26T06:18:44.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2021-07-26T06:18:44.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2021-07-26T06:18:44.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:18:44.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2021-07-26T06:18:44.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2021-07-26T06:18:44.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2021-07-26T06:18:44.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2021-07-26T06:18:44.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2021-07-26T06:18:44.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-07-26T06:18:44.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:18:44.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-07-26T06:18:44.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:18:44.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:18:44.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:18:44.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:18:44.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:18:44.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:18:44.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:18:44.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:18:44.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:18:44.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:18:44.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:18:44.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:18:44.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:18:44.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:18:44.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:18:44.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:18:44.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:18:44.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:18:44.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-07-26T06:18:44.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:18:44.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:44.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:44.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.73043 2021-07-26T06:18:44.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:18:44.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:18:44.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2021-07-26T06:18:44.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:18:44.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:44.491 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:44.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.73043 2021-07-26T06:18:44.491 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:18:44.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:18:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:18:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:18:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2021-07-26T06:18:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2021-07-26T06:18:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2021-07-26T06:18:44.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2021-07-26T06:18:44.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2021-07-26T06:18:44.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:18:44.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:18:44.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:18:44.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2021-07-26T06:18:44.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2021-07-26T06:18:44.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:18:44.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:18:44.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:18:44.885 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:18:44.885 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:44.885 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:44.886 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:18:44.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:18:45.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:18:45.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:18:45.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:18:45.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:18:45.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:18:45.201 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:18:45.202 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:18:45.202 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:18:45.202 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:18:45.202 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:18:45.203 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:45.203 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:45.203 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.73043/ceph-mon.a.asok 2021-07-26T06:18:45.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:18:45.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.73043/ceph-mon.a.asok config get fsid 2021-07-26T06:18:45.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:18:45.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:18:45.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:18:45.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:18:45.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:18:45.295 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:18:45.296 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:18:45.296 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:18:45.296 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:18:45.296 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:45.296 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:45.299 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.73043/ceph-mon.a.asok 2021-07-26T06:18:45.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:18:45.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.73043/ceph-mon.a.asok config get mon_host 2021-07-26T06:18:45.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2021-07-26T06:18:45.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2021-07-26T06:18:45.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:18:45.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:18:45.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:18:45.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2021-07-26T06:18:45.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:18:46.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:18:46.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:18:46.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:18:46.321 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:18:46.321 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:46.321 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:46.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:18:46.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:18:46.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:18:46.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2021-07-26T06:18:46.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-07-26T06:18:46.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:18:46.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:18:46.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:18:46.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2021-07-26T06:18:46.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:18:46.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:18:46.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:18:46.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:18:46.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-07-26T06:18:46.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-07-26T06:18:46.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:18:46.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:18:46.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:18:46.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:18:46.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:18:46.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:18:46.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:18:46.359 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:18:46.359 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:46.359 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:46.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:18:46.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:18:46.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:18:46.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:18:46.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:18:46.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:18:46.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:18:46.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:18:46.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:18:46.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:18:46.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:18:46.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:18:46.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2021-07-26T06:18:46.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:18:46.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=232c3c14-51a4-4169-9ec6-1959ca55b775 2021-07-26T06:18:46.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 232c3c14-51a4-4169-9ec6-1959ca55b775' 2021-07-26T06:18:46.364 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 232c3c14-51a4-4169-9ec6-1959ca55b775 2021-07-26T06:18:46.364 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:18:46.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGU/5gBxGDFRAA9saq4l6YbW1Wqqkhvm+vwQ== 2021-07-26T06:18:46.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGU/5gBxGDFRAA9saq4l6YbW1Wqqkhvm+vwQ=="}' 2021-07-26T06:18:46.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 232c3c14-51a4-4169-9ec6-1959ca55b775 -i td/ec-error-rollforward/0/new.json 2021-07-26T06:18:46.725 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:18:46.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2021-07-26T06:18:46.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGU/5gBxGDFRAA9saq4l6YbW1Wqqkhvm+vwQ== --osd-uuid 232c3c14-51a4-4169-9ec6-1959ca55b775 2021-07-26T06:18:46.778 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:46.753+0000 7fb5b5c29f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:46.779 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:46.757+0000 7fb5b5c29f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:46.779 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:46.757+0000 7fb5b5c29f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:46.779 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:46.757+0000 7fb5b5c29f40 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-07-26T06:18:46.779 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:46.757+0000 7fb5b5c29f40 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-07-26T06:18:46.779 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:46.757+0000 7fb5b5c29f40 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-07-26T06:18:46.891 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:46.873+0000 7fb5b5c29f40 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2021-07-26T06:18:52.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2021-07-26T06:18:52.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:18:52.717 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:18:52.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:18:52.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:18:53.226 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:18:53.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:18:53.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:18:53.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:18:53.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:18:53.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:18:53.255 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:53.236+0000 7fdc94f4bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:53.256 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:53.240+0000 7fdc94f4bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:53.263 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:53.240+0000 7fdc94f4bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:53.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:18:53.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:18:53.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:18:53.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:18:53.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:18:53.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:53.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:18:53.605 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:18:53.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:53.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:18:53.805 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:53.784+0000 7fdc94f4bf40 -1 Falling back to public interface 2021-07-26T06:18:53.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:18:54.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:18:54.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:54.941 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:18:54.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:18:54.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:54.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:18:55.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:18:56.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:18:56.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:56.311 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:18:56.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:18:56.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:56.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:18:56.375 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:56.356+0000 7fdc94f4bf40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:18:56.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:18:57.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:18:57.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:18:57.642 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:18:57.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:18:57.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:18:57.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:18:58.087 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/74147,v1:127.0.0.1:6803/74147] [v2:127.0.0.1:6804/74147,v1:127.0.0.1:6805/74147] exists,up 232c3c14-51a4-4169-9ec6-1959ca55b775 2021-07-26T06:18:58.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:18:58.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:18:58.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:18:58.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2021-07-26T06:18:58.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-07-26T06:18:58.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:18:58.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:18:58.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:18:58.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2021-07-26T06:18:58.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:18:58.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:18:58.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:18:58.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:18:58.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-07-26T06:18:58.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-07-26T06:18:58.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:18:58.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:18:58.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:18:58.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:18:58.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:18:58.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:18:58.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:18:58.098 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:18:58.098 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:18:58.099 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:18:58.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:18:58.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:18:58.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:18:58.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:18:58.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:18:58.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:18:58.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:18:58.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:18:58.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:18:58.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:18:58.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:18:58.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:18:58.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2021-07-26T06:18:58.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:18:58.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0a73d5c-79d1-4b8a-b0ef-f79b4b99df23 2021-07-26T06:18:58.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a0a73d5c-79d1-4b8a-b0ef-f79b4b99df23' 2021-07-26T06:18:58.118 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 a0a73d5c-79d1-4b8a-b0ef-f79b4b99df23 2021-07-26T06:18:58.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:18:58.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSU/5gtceZBRAAuq4zzxkYd3ysNQR0YS6Nqw== 2021-07-26T06:18:58.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSU/5gtceZBRAAuq4zzxkYd3ysNQR0YS6Nqw=="}' 2021-07-26T06:18:58.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0a73d5c-79d1-4b8a-b0ef-f79b4b99df23 -i td/ec-error-rollforward/1/new.json 2021-07-26T06:18:58.592 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:18:58.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2021-07-26T06:18:58.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDSU/5gtceZBRAAuq4zzxkYd3ysNQR0YS6Nqw== --osd-uuid a0a73d5c-79d1-4b8a-b0ef-f79b4b99df23 2021-07-26T06:18:58.644 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:58.620+0000 7f3265faff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:58.650 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:58.624+0000 7f3265faff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:58.651 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:58.624+0000 7f3265faff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:18:58.651 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:58.624+0000 7f3265faff40 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-07-26T06:18:58.651 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:58.624+0000 7f3265faff40 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-07-26T06:18:58.651 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:58.624+0000 7f3265faff40 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-07-26T06:18:58.741 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:18:58.724+0000 7f3265faff40 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2021-07-26T06:19:04.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2021-07-26T06:19:04.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:19:04.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:19:04.826 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:19:04.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:19:05.303 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:19:05.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:19:05.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:05.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:19:05.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:19:05.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:19:05.331 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:05.312+0000 7fe9c373ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:05.336 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:05.320+0000 7fe9c373ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:05.338 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:05.320+0000 7fe9c373ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:05.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:19:05.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:19:05.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:19:05.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:19:05.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:19:05.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:05.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:19:05.751 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:05.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:05.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:19:05.893 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:05.876+0000 7fe9c373ef40 -1 Falling back to public interface 2021-07-26T06:19:06.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:07.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:07.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:07.113 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:19:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:19:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:19:07.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:08.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:08.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:08.494 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:19:08.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:19:08.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:08.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:19:08.523 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:08.496+0000 7fe9c373ef40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:19:08.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:09.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:09.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:09.943 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:19:09.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:19:09.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:09.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:19:10.296 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/75553,v1:127.0.0.1:6811/75553] [v2:127.0.0.1:6812/75553,v1:127.0.0.1:6813/75553] exists,up a0a73d5c-79d1-4b8a-b0ef-f79b4b99df23 2021-07-26T06:19:10.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:19:10.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:19:10.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:19:10.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2021-07-26T06:19:10.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-07-26T06:19:10.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:19:10.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:19:10.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:19:10.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2021-07-26T06:19:10.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:19:10.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:19:10.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:19:10.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:19:10.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-07-26T06:19:10.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-07-26T06:19:10.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:19:10.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:19:10.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:19:10.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:19:10.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:19:10.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:10.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:10.314 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:10.314 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:10.314 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:10.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:10.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:10.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:19:10.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:19:10.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:19:10.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:19:10.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:19:10.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:19:10.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:19:10.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:10.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:19:10.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:19:10.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2021-07-26T06:19:10.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:19:10.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fc2fe65-7889-4b8e-8a29-0f1d0380945a 2021-07-26T06:19:10.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3fc2fe65-7889-4b8e-8a29-0f1d0380945a' 2021-07-26T06:19:10.321 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 3fc2fe65-7889-4b8e-8a29-0f1d0380945a 2021-07-26T06:19:10.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:19:10.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeU/5gRzFHEhAAekmHqWga5YzG2TjrK/p12A== 2021-07-26T06:19:10.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeU/5gRzFHEhAAekmHqWga5YzG2TjrK/p12A=="}' 2021-07-26T06:19:10.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fc2fe65-7889-4b8e-8a29-0f1d0380945a -i td/ec-error-rollforward/2/new.json 2021-07-26T06:19:10.814 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:19:10.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2021-07-26T06:19:10.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDeU/5gRzFHEhAAekmHqWga5YzG2TjrK/p12A== --osd-uuid 3fc2fe65-7889-4b8e-8a29-0f1d0380945a 2021-07-26T06:19:10.855 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:10.836+0000 7fe3cb944f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:10.864 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:10.840+0000 7fe3cb944f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:10.865 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:10.844+0000 7fe3cb944f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:10.865 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:10.844+0000 7fe3cb944f40 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-07-26T06:19:10.866 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:10.844+0000 7fe3cb944f40 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-07-26T06:19:10.866 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:10.844+0000 7fe3cb944f40 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-07-26T06:19:10.979 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:10.964+0000 7fe3cb944f40 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2021-07-26T06:19:17.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2021-07-26T06:19:17.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:19:17.313 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:19:17.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:19:17.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:19:17.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:19:17.794 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:19:17.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:17.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:19:17.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:19:17.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:19:17.822 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:17.807+0000 7fc1a8b97f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:17.825 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:17.811+0000 7fc1a8b97f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:17.826 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:17.811+0000 7fc1a8b97f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:18.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:19:18.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:19:18.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:19:18.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:19:18.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:19:18.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:18.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:19:18.199 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:18.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:18.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:19:18.377 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:18.363+0000 7fc1a8b97f40 -1 Falling back to public interface 2021-07-26T06:19:18.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:19.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:19.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:19.596 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:19:19.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:19:19.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:19.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:19:19.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:20.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:20.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:20.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:19:20.961 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:19:20.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:20.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:19:20.987 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:20.971+0000 7fc1a8b97f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:19:21.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:22.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:22.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:22.421 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:19:22.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:19:22.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:22.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:19:22.827 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/76959,v1:127.0.0.1:6819/76959] [v2:127.0.0.1:6820/76959,v1:127.0.0.1:6821/76959] exists,up 3fc2fe65-7889-4b8e-8a29-0f1d0380945a 2021-07-26T06:19:22.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:19:22.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:19:22.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:19:22.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2021-07-26T06:19:22.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2021-07-26T06:19:22.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:19:22.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:19:22.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:19:22.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2021-07-26T06:19:22.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:19:22.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:19:22.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:19:22.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:19:22.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-07-26T06:19:22.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-07-26T06:19:22.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:19:22.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:19:22.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:19:22.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:19:22.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:19:22.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:22.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:22.839 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:22.839 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:22.839 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:22.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:22.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:22.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:19:22.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:19:22.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:19:22.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:19:22.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:19:22.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:19:22.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:19:22.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:22.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:19:22.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:19:22.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2021-07-26T06:19:22.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:19:22.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03023fa7-9364-4cad-83a0-30a3d844a547 2021-07-26T06:19:22.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 03023fa7-9364-4cad-83a0-30a3d844a547' 2021-07-26T06:19:22.844 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 03023fa7-9364-4cad-83a0-30a3d844a547 2021-07-26T06:19:22.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:19:22.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDqU/5gYw0+MhAAp7RTswk/J+MCJZfbxFhhdg== 2021-07-26T06:19:22.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDqU/5gYw0+MhAAp7RTswk/J+MCJZfbxFhhdg=="}' 2021-07-26T06:19:22.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03023fa7-9364-4cad-83a0-30a3d844a547 -i td/ec-error-rollforward/3/new.json 2021-07-26T06:19:23.373 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:19:23.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2021-07-26T06:19:23.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDqU/5gYw0+MhAAp7RTswk/J+MCJZfbxFhhdg== --osd-uuid 03023fa7-9364-4cad-83a0-30a3d844a547 2021-07-26T06:19:23.420 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:23.407+0000 7fb2c035af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:23.422 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:23.407+0000 7fb2c035af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:23.424 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:23.411+0000 7fb2c035af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:23.425 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:23.411+0000 7fb2c035af40 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-07-26T06:19:23.425 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:23.411+0000 7fb2c035af40 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-07-26T06:19:23.425 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:23.411+0000 7fb2c035af40 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-07-26T06:19:23.532 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:23.515+0000 7fb2c035af40 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2021-07-26T06:19:29.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2021-07-26T06:19:29.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:19:29.964 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T06:19:29.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:19:29.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:19:30.426 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T06:19:30.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:19:30.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:30.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:19:30.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:19:30.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:19:30.452 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:30.431+0000 7f7d015d0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:30.476 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:30.459+0000 7f7d015d0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:30.477 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:30.463+0000 7f7d015d0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:30.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:19:30.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:19:30.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:19:30.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:19:30.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:19:30.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:30.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:19:30.798 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:30.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:30.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:19:31.037 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:31.023+0000 7f7d015d0f40 -1 Falling back to public interface 2021-07-26T06:19:31.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:32.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:32.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:32.140 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:19:32.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:19:32.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:32.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:19:32.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:33.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:33.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:33.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:19:33.515 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:19:33.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:33.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:19:33.559 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:33.543+0000 7f7d015d0f40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:19:33.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:19:34.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:19:34.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:34.934 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:19:34.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:19:34.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:34.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:19:35.333 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78366,v1:127.0.0.1:6827/78366] [v2:127.0.0.1:6828/78366,v1:127.0.0.1:6829/78366] exists,up 03023fa7-9364-4cad-83a0-30a3d844a547 2021-07-26T06:19:35.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:19:35.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:19:35.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:19:35.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2021-07-26T06:19:35.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2021-07-26T06:19:36.675 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'ec' already exists 2021-07-26T06:19:36.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2021-07-26T06:19:38.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2021-07-26T06:19:38.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 76959 2021-07-26T06:19:38.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=78999 2021-07-26T06:19:38.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2021-07-26T06:19:38.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2021-07-26T06:19:39.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 79020' 2021-07-26T06:19:39.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2021-07-26T06:19:39.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 79021' 2021-07-26T06:19:39.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2021-07-26T06:19:39.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 79022' 2021-07-26T06:19:39.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2021-07-26T06:19:39.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2021-07-26T06:19:40.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2021-07-26T06:19:40.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:19:40.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:19:40.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:19:40.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:19:40.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:19:40.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:19:40.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 78999 79020 79021 79022 2021-07-26T06:19:40.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2021-07-26T06:19:40.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2021-07-26T06:19:40.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/ec-error-rollforward 2021-07-26T06:19:40.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:19:40.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:19:40.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:19:40.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/ec-error-rollforward/0 2021-07-26T06:19:40.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:19:40.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:19:40.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:19:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:19:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-07-26T06:19:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-07-26T06:19:40.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:19:40.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:19:40.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:19:40.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:19:40.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:19:40.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:40.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:40.420 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:40.420 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:40.420 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:40.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:40.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:40.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:19:40.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:19:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:19:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:19:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:19:40.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:19:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:19:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/ec-error-rollforward/0 2021-07-26T06:19:40.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:19:40.435 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:19:40.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:40.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/ec-error-rollforward/0/whoami 2021-07-26T06:19:40.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:19:40.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:19:40.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:19:40.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:19:40.441 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:40.427+0000 7f8174429f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:40.454 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:40.439+0000 7f8174429f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:40.457 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:40.443+0000 7f8174429f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:40.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:19:40.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:19:40.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:19:40.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:19:40.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:19:40.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:40.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:19:40.813 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:40.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:40.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:19:40.990 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:40.975+0000 7f8174429f40 -1 Falling back to public interface 2021-07-26T06:19:41.190 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/74147,v1:127.0.0.1:6803/74147] [v2:127.0.0.1:6804/74147,v1:127.0.0.1:6805/74147] exists,up 232c3c14-51a4-4169-9ec6-1959ca55b775 2021-07-26T06:19:41.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:19:41.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:19:41.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:19:41.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2021-07-26T06:19:41.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/ec-error-rollforward 2021-07-26T06:19:41.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:19:41.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:19:41.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:19:41.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/ec-error-rollforward/1 2021-07-26T06:19:41.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:19:41.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:19:41.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:19:41.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:19:41.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-07-26T06:19:41.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-07-26T06:19:41.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:19:41.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:19:41.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:19:41.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:19:41.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:19:41.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:41.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:41.205 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:41.205 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:41.205 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:41.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:41.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:41.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:19:41.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:19:41.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:19:41.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:19:41.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:19:41.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:41.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:19:41.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:19:41.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/ec-error-rollforward/1 2021-07-26T06:19:41.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:19:41.210 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:19:41.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:41.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/ec-error-rollforward/1/whoami 2021-07-26T06:19:41.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:19:41.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:19:41.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:19:41.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:19:41.223 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:41.207+0000 7fd0c045df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:41.227 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:41.211+0000 7fd0c045df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:41.227 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:41.211+0000 7fd0c045df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:41.518 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:41.503+0000 7fd0c045df40 -1 Falling back to public interface 2021-07-26T06:19:41.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:19:41.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:19:41.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:19:41.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:19:41.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:19:41.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:41.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:19:41.605 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:41.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:41.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:19:41.984 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/75553,v1:127.0.0.1:6811/75553] [v2:127.0.0.1:6812/75553,v1:127.0.0.1:6813/75553] exists,up a0a73d5c-79d1-4b8a-b0ef-f79b4b99df23 2021-07-26T06:19:41.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:19:41.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:19:41.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:19:41.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2021-07-26T06:19:41.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/ec-error-rollforward 2021-07-26T06:19:41.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:19:41.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:19:41.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:19:41.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/ec-error-rollforward/2 2021-07-26T06:19:41.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:19:41.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:19:41.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:19:41.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:19:41.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-07-26T06:19:41.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-07-26T06:19:41.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:19:41.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:19:42.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:19:42.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:19:42.000 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:19:42.000 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:42.000 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:42.001 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:42.001 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:42.001 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:42.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:42.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:42.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:19:42.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:19:42.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:19:42.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:19:42.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:19:42.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:42.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:19:42.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:19:42.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/ec-error-rollforward/2 2021-07-26T06:19:42.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:19:42.004 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:19:42.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:42.008 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/ec-error-rollforward/2/whoami 2021-07-26T06:19:42.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:19:42.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:19:42.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:19:42.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:19:42.018 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:41.999+0000 7fc37f03ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:42.018 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:42.003+0000 7fc37f03ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:42.018 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:42.003+0000 7fc37f03ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:42.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:19:42.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:19:42.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:19:42.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:19:42.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:19:42.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:42.389 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:42.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:19:42.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:42.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:19:42.569 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:42.547+0000 7fc37f03ff40 -1 Falling back to public interface 2021-07-26T06:19:42.759 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/76959,v1:127.0.0.1:6819/76959] [v2:127.0.0.1:6820/76959,v1:127.0.0.1:6821/76959] exists,up 3fc2fe65-7889-4b8e-8a29-0f1d0380945a 2021-07-26T06:19:42.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:19:42.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:19:42.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:19:42.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2021-07-26T06:19:42.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/ec-error-rollforward 2021-07-26T06:19:42.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:19:42.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:19:42.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:19:42.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/ec-error-rollforward/3 2021-07-26T06:19:42.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-07-26T06:19:42.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:19:42.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:19:42.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:19:42.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-07-26T06:19:42.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-07-26T06:19:42.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:19:42.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:19:42.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:19:42.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-07-26T06:19:42.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:19:42.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:42.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:42.785 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:42.785 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:42.785 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:42.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:42.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' 2021-07-26T06:19:42.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:19:42.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-07-26T06:19:42.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-07-26T06:19:42.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:19:42.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:19:42.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:42.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:19:42.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:19:42.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/ec-error-rollforward/3 2021-07-26T06:19:42.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:19:42.789 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T06:19:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=d27439d8-3996-4164-953d-07165538618f --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.73043/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:42.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/ec-error-rollforward/3/whoami 2021-07-26T06:19:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:19:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:19:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:19:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:19:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:42.775+0000 7f432f8b9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:42.775+0000 7f432f8b9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:42.793 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:42.779+0000 7f432f8b9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:43.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:19:43.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:19:43.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:19:43.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:19:43.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:19:43.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:19:43.197 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:43.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:19:43.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:19:43.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:19:43.335 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:43.318+0000 7f432f8b9f40 -1 Falling back to public interface 2021-07-26T06:19:43.711 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/78366,v1:127.0.0.1:6827/78366] [v2:127.0.0.1:6828/78366,v1:127.0.0.1:6829/78366] exists,up 03023fa7-9364-4cad-83a0-30a3d844a547 2021-07-26T06:19:43.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:19:43.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:19:43.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:19:43.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2021-07-26T06:19:43.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:19:43.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:19:43.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:19:43.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:19:43.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:19:43.715 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:19:43.716 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:19:43.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:19:43.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:19:43.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:19:43.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:19:43.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:19:43.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:19:43.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:19:43.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:19:44.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:19:44.288 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:19:44.288 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:19:44.288 INFO:tasks.workunit.client.0.smithi094.stderr:3' 2021-07-26T06:19:44.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:19:44.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:19:44.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:19:45.196 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:45.182+0000 7f8174429f40 -1 osd.0 25 log_to_monitors {default=true} 2021-07-26T06:19:45.545 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:45.530+0000 7fd0c045df40 -1 osd.1 25 log_to_monitors {default=true} 2021-07-26T06:19:46.792 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:46.774+0000 7fc37f03ff40 -1 osd.2 24 log_to_monitors {default=true} 2021-07-26T06:19:47.586 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:47.558+0000 7f432f8b9f40 -1 osd.3 25 log_to_monitors {default=true} 2021-07-26T06:19:49.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=120259084291 2021-07-26T06:19:49.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 120259084291 2021-07-26T06:19:49.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084291' 2021-07-26T06:19:49.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:19:49.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:19:49.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018883 2021-07-26T06:19:49.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018883 2021-07-26T06:19:49.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084291 1-128849018883' 2021-07-26T06:19:49.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:19:49.639 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:19:49.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=141733920771 2021-07-26T06:19:49.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 141733920771 2021-07-26T06:19:49.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084291 1-128849018883 2-141733920771' 2021-07-26T06:19:49.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:19:49.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:19:49.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=146028888067 2021-07-26T06:19:49.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 146028888067 2021-07-26T06:19:49.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-120259084291 1-128849018883 2-141733920771 3-146028888067' 2021-07-26T06:19:49.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:19:49.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-120259084291 2021-07-26T06:19:49.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:19:49.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:19:49.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-120259084291 2021-07-26T06:19:49.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:19:49.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=120259084291 2021-07-26T06:19:49.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2021-07-26T06:19:49.918 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 120259084291 2021-07-26T06:19:49.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:19:50.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 1 -lt 120259084291 2021-07-26T06:19:50.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:19:51.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:19:51.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:19:51.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 120259084291 -lt 120259084291 2021-07-26T06:19:51.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:19:51.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-128849018883 2021-07-26T06:19:51.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:19:51.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:19:51.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-128849018883 2021-07-26T06:19:51.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:19:51.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018883 2021-07-26T06:19:51.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 128849018883' 2021-07-26T06:19:51.675 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 128849018883 2021-07-26T06:19:51.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:19:52.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018883 -lt 128849018883 2021-07-26T06:19:52.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:19:52.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-141733920771 2021-07-26T06:19:52.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:19:52.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:19:52.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-141733920771 2021-07-26T06:19:52.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:19:52.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=141733920771 2021-07-26T06:19:52.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2021-07-26T06:19:52.034 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 141733920771 2021-07-26T06:19:52.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:19:52.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 141733920771 -lt 141733920771 2021-07-26T06:19:52.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:19:52.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-146028888067 2021-07-26T06:19:52.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:19:52.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:19:52.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-146028888067 2021-07-26T06:19:52.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:19:52.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=146028888067 2021-07-26T06:19:52.403 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 146028888067 2021-07-26T06:19:52.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 146028888067' 2021-07-26T06:19:52.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:19:52.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 146028888067 -lt 146028888067 2021-07-26T06:19:52.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:19:52.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:19:52.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:19:53.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T06:19:53.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:19:53.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:19:53.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:19:53.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:19:53.265 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:19:53.265 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:19:53.266 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:19:53.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:19:53.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:19:53.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:19:53.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:19:54.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T06:19:54.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:19:54.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:19:54.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2021-07-26T06:19:54.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-07-26T06:19:54.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:19:54.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-07-26T06:19:54.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:19:54.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:19:54.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:19:54.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:19:54.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:19:54.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:19:54.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:19:54.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:19:54.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:19:54.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:19:54.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:19:54.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:19:54.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:19:54.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:19:54.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:19:54.345 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:19:54.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:19:54.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:19:54.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-07-26T06:19:54.346 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:19:54.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:54.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:54.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.73043 2021-07-26T06:19:54.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:19:54.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:19:54.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:19:54.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/ec-error-rollforward 0 2021-07-26T06:19:54.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-07-26T06:19:54.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:19:54.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-07-26T06:19:54.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:19:54.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:19:54.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:19:54.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:19:54.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:19:54.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:19:54.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:19:54.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:19:54.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:19:54.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:19:54.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:19:54.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:19:54.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:19:54.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:19:54.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:19:54.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:19:54.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:19:54.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:19:54.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-07-26T06:19:54.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:19:54.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:54.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.73043 2021-07-26T06:19:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.73043 2021-07-26T06:19:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:19:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:19:54.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:19:54.374 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:19:54.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:19:54.400 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2021-07-26T06:19:54.400 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2021-07-26T06:19:54.585 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:19:54.585 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:19:54.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:19:54.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:19:54.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:19:54.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:19:54.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:19:54.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:19:54.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:19:54.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:19:54.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-bench 2021-07-26T06:19:54.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2021-07-26T06:19:54.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2021-07-26T06:19:54.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2021-07-26T06:19:54.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2021-07-26T06:19:54.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2021-07-26T06:19:54.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2021-07-26T06:19:54.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=b7a2b6f6-10a9-4a3f-84a7-9a90a81ca13f --auth-supported=none ' 2021-07-26T06:19:54.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2021-07-26T06:19:54.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2021-07-26T06:19:54.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2021-07-26T06:19:54.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:19:54.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2021-07-26T06:19:54.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2021-07-26T06:19:54.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2021-07-26T06:19:54.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2021-07-26T06:19:54.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2021-07-26T06:19:54.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-07-26T06:19:54.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:19:54.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-07-26T06:19:54.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:19:54.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:19:54.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:19:54.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:19:54.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:19:54.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:19:54.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:19:54.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:19:54.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:19:54.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:19:54.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:19:54.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:19:54.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:19:54.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:19:54.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:19:54.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:19:54.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:19:54.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:19:54.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-07-26T06:19:54.646 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:19:54.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:54.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:19:54.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81614 2021-07-26T06:19:54.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:19:54.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:19:54.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2021-07-26T06:19:54.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:19:54.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:54.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:19:54.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81614 2021-07-26T06:19:54.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:19:54.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:19:54.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:19:54.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:19:54.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2021-07-26T06:19:54.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2021-07-26T06:19:54.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2021-07-26T06:19:54.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2021-07-26T06:19:54.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2021-07-26T06:19:54.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:19:54.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:19:54.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:19:54.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2021-07-26T06:19:54.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2021-07-26T06:19:54.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:19:54.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:54.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:54.952 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:54.953 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:54.953 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:19:54.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81614/$cluster-$name.asok' 2021-07-26T06:19:55.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.81614/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:19:55.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:19:55.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:19:55.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:19:55.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:19:55.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:19:55.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:19:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:19:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:19:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:19:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:19:55.277 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:55.277 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:19:55.277 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81614/ceph-mon.a.asok 2021-07-26T06:19:55.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:19:55.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.81614/ceph-mon.a.asok config get fsid 2021-07-26T06:19:55.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:19:55.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:19:55.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:19:55.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:19:55.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:19:55.375 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:19:55.376 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:19:55.377 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:19:55.377 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:19:55.378 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:55.379 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:19:55.379 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81614/ceph-mon.a.asok 2021-07-26T06:19:55.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:19:55.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.81614/ceph-mon.a.asok config get mon_host 2021-07-26T06:19:55.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2021-07-26T06:19:55.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2021-07-26T06:19:55.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:19:55.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:19:55.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:19:55.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2021-07-26T06:19:55.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:19:56.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:19:56.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:56.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:56.343 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:56.344 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:56.344 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:19:56.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81614/$cluster-$name.asok' 2021-07-26T06:19:56.344 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:19:56.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.81614/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:19:56.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2021-07-26T06:19:56.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2021-07-26T06:19:56.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:19:56.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:19:56.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:19:56.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2021-07-26T06:19:56.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b7a2b6f6-10a9-4a3f-84a7-9a90a81ca13f --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2021-07-26T06:19:56.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:19:56.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:19:56.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:19:56.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2021-07-26T06:19:56.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2021-07-26T06:19:56.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:19:56.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:19:56.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:19:56.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2021-07-26T06:19:56.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:19:56.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:19:56.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:19:56.402 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:19:56.402 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:19:56.402 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:19:56.403 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81614/$cluster-$name.asok' 2021-07-26T06:19:56.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81614/$cluster-$name.asok' 2021-07-26T06:19:56.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:19:56.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:19:56.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:19:56.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2021-07-26T06:19:56.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2021-07-26T06:19:56.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:19:56.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:19:56.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:19:56.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:19:56.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:19:56.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2021-07-26T06:19:56.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:19:56.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc4ad2a2-50a1-4b29-adf2-1d1e39c5b211 2021-07-26T06:19:56.410 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 fc4ad2a2-50a1-4b29-adf2-1d1e39c5b211 2021-07-26T06:19:56.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fc4ad2a2-50a1-4b29-adf2-1d1e39c5b211' 2021-07-26T06:19:56.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:19:56.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMVP5g+nZ2FhAA+D72R4dCPKPkx7vD8vFK8w== 2021-07-26T06:19:56.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMVP5g+nZ2FhAA+D72R4dCPKPkx7vD8vFK8w=="}' 2021-07-26T06:19:56.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc4ad2a2-50a1-4b29-adf2-1d1e39c5b211 -i td/osd-bench/0/new.json 2021-07-26T06:19:56.803 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:19:56.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2021-07-26T06:19:56.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b7a2b6f6-10a9-4a3f-84a7-9a90a81ca13f --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.81614/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAMVP5g+nZ2FhAA+D72R4dCPKPkx7vD8vFK8w== --osd-uuid fc4ad2a2-50a1-4b29-adf2-1d1e39c5b211 2021-07-26T06:19:56.856 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:56.826+0000 7fbd05ed2f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:56.856 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:56.826+0000 7fbd05ed2f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:56.857 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:56.830+0000 7fbd05ed2f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:19:56.857 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:56.830+0000 7fbd05ed2f40 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-07-26T06:19:56.857 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:56.830+0000 7fbd05ed2f40 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-07-26T06:19:56.857 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:56.830+0000 7fbd05ed2f40 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-07-26T06:19:56.933 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:19:56.910+0000 7fbd05ed2f40 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2021-07-26T06:20:02.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2021-07-26T06:20:02.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:20:02.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:20:02.858 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:20:02.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:20:03.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:20:03.336 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:20:03.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b7a2b6f6-10a9-4a3f-84a7-9a90a81ca13f --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.81614/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:20:03.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:20:03.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:20:03.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:20:03.346 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:20:03.330+0000 7fb02eb57f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:20:03.374 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:20:03.358+0000 7fb02eb57f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:20:03.377 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:20:03.358+0000 7fb02eb57f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:20:03.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:20:03.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:20:03.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:20:03.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:20:03.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:20:03.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:20:03.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:20:03.676 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:20:03.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:20:03.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:20:03.911 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:20:03.894+0000 7fb02eb57f40 -1 Falling back to public interface 2021-07-26T06:20:03.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:20:04.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:20:04.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:20:04.993 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:20:04.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:20:04.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:20:04.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:20:05.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:20:06.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:20:06.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:20:06.331 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:20:06.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:20:06.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:20:06.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:20:06.481 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:20:06.466+0000 7fb02eb57f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:20:06.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:20:07.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:20:07.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:20:07.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:20:07.802 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:20:07.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:20:07.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:20:08.195 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/82719,v1:127.0.0.1:6803/82719] [v2:127.0.0.1:6804/82719,v1:127.0.0.1:6805/82719] exists,up fc4ad2a2-50a1-4b29-adf2-1d1e39c5b211 2021-07-26T06:20:08.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:20:08.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:20:08.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:20:08.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2021-07-26T06:20:08.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2021-07-26T06:20:08.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2021-07-26T06:20:08.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2021-07-26T06:20:08.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2021-07-26T06:20:08.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2021-07-26T06:20:08.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2021-07-26T06:20:08.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2021-07-26T06:20:08.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2021-07-26T06:20:08.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2021-07-26T06:20:08.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2021-07-26T06:20:08.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2021-07-26T06:20:08.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2021-07-26T06:20:08.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local dir=td/osd-bench 2021-07-26T06:20:08.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: shift 2021-07-26T06:20:08.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: local expected=osd_bench_max_block_size 2021-07-26T06:20:08.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: shift 2021-07-26T06:20:08.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: expect_failure: local success 2021-07-26T06:20:08.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: ceph tell osd.0 bench 1024 67108865 2021-07-26T06:20:08.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: success=false 2021-07-26T06:20:08.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: false 2021-07-26T06:20:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2021-07-26T06:20:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: expect_failure: return 0 2021-07-26T06:20:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2021-07-26T06:20:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2021-07-26T06:20:08.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2021-07-26T06:20:08.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local dir=td/osd-bench 2021-07-26T06:20:08.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: shift 2021-07-26T06:20:08.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: local expected=bench_small_size_max_iops 2021-07-26T06:20:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: shift 2021-07-26T06:20:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: expect_failure: local success 2021-07-26T06:20:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: ceph tell osd.0 bench 3072001 1024 2021-07-26T06:20:08.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: success=false 2021-07-26T06:20:08.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: false 2021-07-26T06:20:08.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2021-07-26T06:20:08.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: expect_failure: return 0 2021-07-26T06:20:08.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2021-07-26T06:20:08.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2021-07-26T06:20:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2021-07-26T06:20:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local dir=td/osd-bench 2021-07-26T06:20:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1837: expect_failure: shift 2021-07-26T06:20:08.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: local expected=osd_bench_large_size_max_throughput 2021-07-26T06:20:08.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: expect_failure: shift 2021-07-26T06:20:08.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: expect_failure: local success 2021-07-26T06:20:08.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2021-07-26T06:20:08.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1845: expect_failure: success=false 2021-07-26T06:20:08.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: false 2021-07-26T06:20:08.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2021-07-26T06:20:08.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: expect_failure: return 0 2021-07-26T06:20:08.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2021-07-26T06:21:19.836 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:21:19.837 INFO:tasks.workunit.client.0.smithi094.stdout: "bytes_written": 1073741824, 2021-07-26T06:21:19.860 INFO:tasks.workunit.client.0.smithi094.stdout: "blocksize": 4194304, 2021-07-26T06:21:19.861 INFO:tasks.workunit.client.0.smithi094.stdout: "elapsed_sec": 37.647370766999998, 2021-07-26T06:21:19.861 INFO:tasks.workunit.client.0.smithi094.stdout: "bytes_per_sec": 28521030.874782737, 2021-07-26T06:21:19.861 INFO:tasks.workunit.client.0.smithi094.stdout: "iops": 6.7999436556774944 2021-07-26T06:21:19.861 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:21:19.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2021-07-26T06:21:20.525 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:21:20.525 INFO:tasks.workunit.client.0.smithi094.stdout: "bytes_written": 10, 2021-07-26T06:21:20.525 INFO:tasks.workunit.client.0.smithi094.stdout: "blocksize": 14456, 2021-07-26T06:21:20.525 INFO:tasks.workunit.client.0.smithi094.stdout: "elapsed_sec": 0.059237906999999999, 2021-07-26T06:21:20.526 INFO:tasks.workunit.client.0.smithi094.stdout: "bytes_per_sec": 168.81082581124954, 2021-07-26T06:21:20.526 INFO:tasks.workunit.client.0.smithi094.stdout: "iops": 0.011677561276373101 2021-07-26T06:21:20.526 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:21:20.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2021-07-26T06:21:20.891 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:21:20.892 INFO:tasks.workunit.client.0.smithi094.stdout: "bytes_written": 1, 2021-07-26T06:21:20.892 INFO:tasks.workunit.client.0.smithi094.stdout: "blocksize": 14456, 2021-07-26T06:21:20.895 INFO:tasks.workunit.client.0.smithi094.stdout: "elapsed_sec": 0.048993269999999998, 2021-07-26T06:21:20.896 INFO:tasks.workunit.client.0.smithi094.stdout: "bytes_per_sec": 20.410966649092824, 2021-07-26T06:21:20.896 INFO:tasks.workunit.client.0.smithi094.stdout: "iops": 0.0014119373719627022 2021-07-26T06:21:20.896 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:21:20.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2021-07-26T06:21:20.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-07-26T06:21:20.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:21:20.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-07-26T06:21:20.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:21:20.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:21:20.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:21:20.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:21:20.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:21:21.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:21:21.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:21:21.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:21:21.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:21:21.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:21:21.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:21:21.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:21:21.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:21:21.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:21:21.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:21:21.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:21:21.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:21:21.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:21:21.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-07-26T06:21:21.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:21:21.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:21.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:21:21.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81614 2021-07-26T06:21:21.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:21:21.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:21:21.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:21:21.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-bench 0 2021-07-26T06:21:21.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-07-26T06:21:21.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:21:21.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-07-26T06:21:21.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:21:21.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:21:21.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:21:21.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:21:21.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:21:21.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:21:21.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:21:21.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:21:21.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:21:21.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:21:21.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:21:21.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:21:21.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:21:21.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:21:21.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:21:21.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:21:21.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:21:21.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:21:21.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-07-26T06:21:21.458 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:21:21.458 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:21.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81614 2021-07-26T06:21:21.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81614 2021-07-26T06:21:21.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:21:21.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:21:21.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:21:21.462 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:21:21.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:21:21.511 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2021-07-26T06:21:21.512 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2021-07-26T06:21:21.950 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:21:21.950 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:21:22.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:21:22.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:21:22.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:21:22.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:21:22.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:21:22.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:21:22.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:21:22.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:21:22.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-bluefs-volume-ops 2021-07-26T06:21:22.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:22.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2021-07-26T06:21:22.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-07-26T06:21:22.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-07-26T06:21:22.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: export CEPH_ARGS 2021-07-26T06:21:22.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: uuidgen 2021-07-26T06:21:22.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: CEPH_ARGS+='--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none ' 2021-07-26T06:21:22.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-07-26T06:21:22.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2021-07-26T06:21:22.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:16: run: CEPH_ARGS+='--bluestore_block_db_create=true ' 2021-07-26T06:21:22.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:17: run: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2021-07-26T06:21:22.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:18: run: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2021-07-26T06:21:22.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:19: run: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2021-07-26T06:21:22.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: run: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2021-07-26T06:21:22.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: set 2021-07-26T06:21:22.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:21:22.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: local funcs=TEST_bluestore 2021-07-26T06:21:22.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: run: for func in $funcs 2021-07-26T06:21:22.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: run: setup td/osd-bluefs-volume-ops 2021-07-26T06:21:22.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:22.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2021-07-26T06:21:22.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:22.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:21:22.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-07-26T06:21:22.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:21:22.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:21:22.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:21:22.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:21:22.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:21:22.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:21:22.243 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:21:22.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:21:22.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:21:22.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:21:22.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:21:22.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:21:22.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:21:22.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:21:22.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:21:22.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:21:22.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:21:22.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:21:22.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-07-26T06:21:22.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:21:22.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:22.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:22.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83476 2021-07-26T06:21:22.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:21:22.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:21:22.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2021-07-26T06:21:22.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:21:22.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:22.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:22.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83476 2021-07-26T06:21:22.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:21:22.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:21:22.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:21:22.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:21:22.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2021-07-26T06:21:22.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:24: run: TEST_bluestore td/osd-bluefs-volume-ops 2021-07-26T06:21:22.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:22.256 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: ulimit -n 2021-07-26T06:21:22.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: local flimit=4096 2021-07-26T06:21:22.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: '[' 4096 -lt 1536 ']' 2021-07-26T06:21:22.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2021-07-26T06:21:22.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:22.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:21:22.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:21:22.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:21:22.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2021-07-26T06:21:22.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:22.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:21:22.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:21:22.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:21:22.271 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:21:22.271 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:22.272 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:22.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:21:22.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:21:22.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:21:22.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:21:22.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:21:22.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:21:22.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:21:22.349 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:21:22.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:21:22.368 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:21:22.368 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:21:22.378 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:21:22.378 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:22.379 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:22.379 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83476/ceph-mon.a.asok 2021-07-26T06:21:22.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:21:22.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.83476/ceph-mon.a.asok config get fsid 2021-07-26T06:21:22.408 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:21:22.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:21:22.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:21:22.431 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:21:22.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:21:22.560 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:21:22.560 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:21:22.561 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:21:22.647 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:21:22.648 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:22.648 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:22.648 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83476/ceph-mon.a.asok 2021-07-26T06:21:22.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:21:22.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.83476/ceph-mon.a.asok config get mon_host 2021-07-26T06:21:22.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2021-07-26T06:21:22.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:22.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:21:22.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:21:22.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:21:22.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2021-07-26T06:21:22.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:21:23.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:21:23.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:21:23.495 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:21:23.505 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:21:23.509 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:23.509 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:23.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:21:23.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:21:23.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:21:23.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2021-07-26T06:21:23.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:23.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:21:23.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:21:23.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:21:23.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-07-26T06:21:23.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:21:23.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:21:23.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:21:23.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:21:23.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-07-26T06:21:23.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-07-26T06:21:23.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:21:23.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:21:23.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:21:23.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:21:23.735 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:21:23.735 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:21:23.735 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:21:23.735 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:21:23.736 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:23.736 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:23.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:21:23.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:21:23.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:21:23.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:21:23.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:21:23.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:21:23.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:21:23.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:21:23.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:21:23.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:21:23.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:21:23.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:21:23.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-07-26T06:21:23.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:21:23.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=48983ae5-ea84-4702-9a7d-c6e04ee74555 2021-07-26T06:21:23.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 48983ae5-ea84-4702-9a7d-c6e04ee74555' 2021-07-26T06:21:23.868 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 48983ae5-ea84-4702-9a7d-c6e04ee74555 2021-07-26T06:21:23.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:21:23.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjVP5gjmKJGxAAux5xqCUsT6v/Tc6lmiBb4Q== 2021-07-26T06:21:23.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjVP5gjmKJGxAAux5xqCUsT6v/Tc6lmiBb4Q=="}' 2021-07-26T06:21:23.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 48983ae5-ea84-4702-9a7d-c6e04ee74555 -i td/osd-bluefs-volume-ops/0/new.json 2021-07-26T06:21:23.870 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:21:23.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2021-07-26T06:21:23.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBjVP5gjmKJGxAAux5xqCUsT6v/Tc6lmiBb4Q== --osd-uuid 48983ae5-ea84-4702-9a7d-c6e04ee74555 2021-07-26T06:21:23.877 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:23.858+0000 7fb1a61d0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:23.964 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:23.862+0000 7fb1a61d0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:23.985 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:23.862+0000 7fb1a61d0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:23.985 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:23.862+0000 7fb1a61d0f40 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-07-26T06:21:23.986 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:23.862+0000 7fb1a61d0f40 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-07-26T06:21:23.986 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:23.862+0000 7fb1a61d0f40 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-07-26T06:21:24.041 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:23.986+0000 7fb1a61d0f40 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2021-07-26T06:21:33.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2021-07-26T06:21:33.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:21:33.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:21:33.886 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:21:33.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:21:34.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:21:34.254 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:21:34.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:21:34.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:21:34.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:21:34.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:21:34.286 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:34.218+0000 7f29af379f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:34.286 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:34.222+0000 7f29af379f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:34.286 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:34.222+0000 7f29af379f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:34.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:21:34.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:21:34.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:21:34.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:21:34.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:21:34.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:34.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:21:34.666 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:21:34.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:34.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:21:34.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:35.274 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:35.254+0000 7f29af379f40 -1 Falling back to public interface 2021-07-26T06:21:36.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:36.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:36.020 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:21:36.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:21:36.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:21:36.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:36.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:37.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:37.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:37.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:21:37.523 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:21:37.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:37.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:21:37.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:38.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:38.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:38.689 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:21:38.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:21:38.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:38.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:21:39.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:40.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:40.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:40.040 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:21:40.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:21:40.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:40.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:21:40.203 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:40.150+0000 7f29af379f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:21:40.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:41.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:41.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:41.573 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:21:41.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:21:41.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:41.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:21:41.980 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/84977,v1:127.0.0.1:6803/84977] [v2:127.0.0.1:6804/84977,v1:127.0.0.1:6805/84977] exists,up 48983ae5-ea84-4702-9a7d-c6e04ee74555 2021-07-26T06:21:42.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:21:42.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:21:42.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:21:42.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-07-26T06:21:42.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=84977 2021-07-26T06:21:42.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2021-07-26T06:21:42.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:21:42.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:21:42.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:21:42.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:21:42.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-07-26T06:21:42.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:21:42.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:21:42.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:21:42.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:21:42.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-07-26T06:21:42.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-07-26T06:21:42.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:21:42.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:21:42.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:21:42.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:21:42.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:21:42.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:21:42.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:21:42.209 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:21:42.209 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:21:42.209 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:21:42.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:21:42.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:21:42.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:21:42.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:21:42.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:21:42.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:21:42.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:21:42.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:21:42.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:21:42.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:21:42.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:21:42.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:21:42.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-07-26T06:21:42.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:21:42.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=817d3194-fc16-4662-9b83-ea4d7aabf413 2021-07-26T06:21:42.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 817d3194-fc16-4662-9b83-ea4d7aabf413' 2021-07-26T06:21:42.434 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 817d3194-fc16-4662-9b83-ea4d7aabf413 2021-07-26T06:21:42.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:21:42.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1VP5gZQ0eORAAHWz0zopeSlR2X9pN/7ApPw== 2021-07-26T06:21:42.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1VP5gZQ0eORAAHWz0zopeSlR2X9pN/7ApPw=="}' 2021-07-26T06:21:42.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 817d3194-fc16-4662-9b83-ea4d7aabf413 -i td/osd-bluefs-volume-ops/1/new.json 2021-07-26T06:21:42.737 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:21:42.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2021-07-26T06:21:42.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB1VP5gZQ0eORAAHWz0zopeSlR2X9pN/7ApPw== --osd-uuid 817d3194-fc16-4662-9b83-ea4d7aabf413 2021-07-26T06:21:42.753 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:42.501+0000 7f7204402f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:42.753 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:42.505+0000 7f7204402f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:42.753 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:42.505+0000 7f7204402f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:42.753 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:42.505+0000 7f7204402f40 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-07-26T06:21:42.754 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:42.505+0000 7f7204402f40 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-07-26T06:21:42.754 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:42.505+0000 7f7204402f40 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-07-26T06:21:42.754 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:42.597+0000 7f7204402f40 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2021-07-26T06:21:52.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2021-07-26T06:21:52.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:21:52.764 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:21:52.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:21:52.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:21:53.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:21:53.291 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:21:53.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:21:53.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:21:53.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:21:53.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:21:53.325 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:53.257+0000 7f9127b89f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:53.325 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:53.257+0000 7f9127b89f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:53.325 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:53.261+0000 7f9127b89f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:21:53.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:21:53.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:21:53.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:21:53.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:21:53.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:21:53.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:53.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:21:53.698 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:21:53.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:53.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:21:54.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:54.330 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:54.309+0000 7f9127b89f40 -1 Falling back to public interface 2021-07-26T06:21:55.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:55.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:55.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:21:55.049 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:21:55.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:55.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:21:55.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:56.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:56.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:56.385 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:21:56.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:21:56.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:56.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:21:56.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:57.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:57.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:57.758 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:21:57.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:21:57.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:57.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:21:58.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:21:59.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:21:59.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:21:59.151 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:21:59.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:21:59.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:21:59.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:21:59.225 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:21:59.201+0000 7f9127b89f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:21:59.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:00.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:00.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:00.624 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:22:00.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:22:00.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:00.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:22:00.966 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/87218,v1:127.0.0.1:6811/87218] [v2:127.0.0.1:6812/87218,v1:127.0.0.1:6813/87218] exists,up 817d3194-fc16-4662-9b83-ea4d7aabf413 2021-07-26T06:22:01.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:22:01.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:22:01.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:22:01.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-07-26T06:22:01.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=87218 2021-07-26T06:22:01.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2021-07-26T06:22:01.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:22:01.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:22:01.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:22:01.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:22:01.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-07-26T06:22:01.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:22:01.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:22:01.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:22:01.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:22:01.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-07-26T06:22:01.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-07-26T06:22:01.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:22:01.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:22:01.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:22:01.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:22:01.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:22:01.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:22:01.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:22:01.185 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:22:01.185 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:22:01.186 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:22:01.237 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:22:01.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:22:01.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:22:01.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:22:01.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:22:01.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:22:01.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:22:01.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:22:01.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:22:01.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:22:01.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:22:01.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:22:01.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-07-26T06:22:01.256 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:22:01.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47d572b5-8285-4254-ab35-52b7fa3c0ddf 2021-07-26T06:22:01.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 47d572b5-8285-4254-ab35-52b7fa3c0ddf' 2021-07-26T06:22:01.538 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 47d572b5-8285-4254-ab35-52b7fa3c0ddf 2021-07-26T06:22:01.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:22:01.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIVP5ghas+OhAAajLmeGTjdE7flrjvqgl/Xw== 2021-07-26T06:22:01.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIVP5ghas+OhAAajLmeGTjdE7flrjvqgl/Xw=="}' 2021-07-26T06:22:01.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47d572b5-8285-4254-ab35-52b7fa3c0ddf -i td/osd-bluefs-volume-ops/2/new.json 2021-07-26T06:22:01.544 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:22:01.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2021-07-26T06:22:01.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCIVP5ghas+OhAAajLmeGTjdE7flrjvqgl/Xw== --osd-uuid 47d572b5-8285-4254-ab35-52b7fa3c0ddf 2021-07-26T06:22:01.545 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:01.521+0000 7f3b8a4c8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:01.958 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:01.565+0000 7f3b8a4c8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:02.071 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:01.565+0000 7f3b8a4c8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:02.072 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:01.565+0000 7f3b8a4c8f40 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-07-26T06:22:02.072 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:01.565+0000 7f3b8a4c8f40 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-07-26T06:22:02.072 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:01.565+0000 7f3b8a4c8f40 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-07-26T06:22:02.072 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:01.653+0000 7f3b8a4c8f40 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2021-07-26T06:22:11.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2021-07-26T06:22:11.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:22:11.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:22:11.897 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:22:11.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:22:12.357 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:22:12.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:22:12.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:22:12.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:22:12.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:22:12.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:22:12.450 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:12.328+0000 7f68d54cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:12.450 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:12.332+0000 7f68d54cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:12.450 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:12.332+0000 7f68d54cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:12.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:22:12.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:22:12.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:22:12.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:22:12.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:22:12.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:12.784 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:22:12.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:22:12.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:12.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:22:13.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:13.153 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:13.120+0000 7f68d54cff40 -1 Falling back to public interface 2021-07-26T06:22:14.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:14.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:14.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:22:14.152 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:22:14.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:14.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:22:14.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:15.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:15.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:15.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:22:15.483 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:22:15.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:15.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:22:15.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:16.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:16.857 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:22:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:22:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:16.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:22:17.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:18.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:18.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:18.253 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:22:18.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:22:18.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:18.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:22:18.304 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:18.220+0000 7f68d54cff40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:22:18.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:19.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:19.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:19.610 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:22:19.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:22:19.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:19.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:22:20.028 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/89459,v1:127.0.0.1:6819/89459] [v2:127.0.0.1:6820/89459,v1:127.0.0.1:6821/89459] exists,up 47d572b5-8285-4254-ab35-52b7fa3c0ddf 2021-07-26T06:22:20.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:22:20.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:22:20.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:22:20.441 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-07-26T06:22:20.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=89459 2021-07-26T06:22:20.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2021-07-26T06:22:20.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:22:20.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:22:20.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:22:20.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:22:20.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-07-26T06:22:20.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:22:20.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:22:20.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:22:20.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:22:20.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-07-26T06:22:20.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-07-26T06:22:20.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:22:20.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:22:20.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:22:20.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:22:20.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:22:20.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:22:20.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:22:20.703 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:22:20.703 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:22:20.703 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:22:20.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:22:20.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:22:20.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:22:20.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:22:20.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:22:20.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:22:20.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:22:20.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:22:20.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:22:20.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:22:20.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:22:20.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:22:20.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-07-26T06:22:20.803 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:22:20.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93101700-5763-487e-b234-f4cc2d81e938 2021-07-26T06:22:20.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 93101700-5763-487e-b234-f4cc2d81e938' 2021-07-26T06:22:20.828 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 93101700-5763-487e-b234-f4cc2d81e938 2021-07-26T06:22:20.841 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:22:20.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcVP5gloyrABAAye8qRh2yXoGaOK0gxmGv0A== 2021-07-26T06:22:20.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcVP5gloyrABAAye8qRh2yXoGaOK0gxmGv0A=="}' 2021-07-26T06:22:20.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93101700-5763-487e-b234-f4cc2d81e938 -i td/osd-bluefs-volume-ops/3/new.json 2021-07-26T06:22:20.879 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:22:20.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2021-07-26T06:22:21.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCcVP5gloyrABAAye8qRh2yXoGaOK0gxmGv0A== --osd-uuid 93101700-5763-487e-b234-f4cc2d81e938 2021-07-26T06:22:21.011 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:20.604+0000 7f85b0f37f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:21.011 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:20.608+0000 7f85b0f37f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:21.011 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:20.612+0000 7f85b0f37f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:21.011 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:20.612+0000 7f85b0f37f40 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-07-26T06:22:21.012 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:20.612+0000 7f85b0f37f40 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-07-26T06:22:21.012 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:20.612+0000 7f85b0f37f40 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-07-26T06:22:21.012 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:20.708+0000 7f85b0f37f40 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2021-07-26T06:22:31.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2021-07-26T06:22:31.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:22:31.243 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T06:22:31.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:22:31.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:22:31.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:22:31.714 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T06:22:31.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:22:31.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:22:31.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:22:31.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:22:31.748 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:31.723+0000 7fb6e4bb8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:31.768 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:31.723+0000 7fb6e4bb8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:31.782 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:31.727+0000 7fb6e4bb8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:22:32.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:22:32.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:22:32.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:22:32.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:22:32.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:22:32.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:32.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:22:32.061 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:22:32.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:32.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:22:32.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:32.783 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:32.759+0000 7fb6e4bb8f40 -1 Falling back to public interface 2021-07-26T06:22:33.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:33.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:33.453 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:22:33.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:22:33.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:33.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:22:33.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:34.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:34.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:34.834 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:22:34.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:22:34.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:34.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:22:35.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:36.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:36.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:36.176 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:22:36.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:22:36.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:22:36.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:36.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:37.427 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:22:37.399+0000 7fb6e4bb8f40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:22:37.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:37.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:37.536 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:22:37.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:22:37.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:37.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:22:38.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:22:39.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:22:39.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:22:39.099 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:22:39.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:22:39.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:22:39.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:22:39.485 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/91700,v1:127.0.0.1:6827/91700] [v2:127.0.0.1:6828/91700,v1:127.0.0.1:6829/91700] exists,up 93101700-5763-487e-b234-f4cc2d81e938 2021-07-26T06:22:39.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:22:39.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:22:39.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:22:39.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-07-26T06:22:39.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=91700 2021-07-26T06:22:39.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2021-07-26T06:22:44.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2021-07-26T06:22:44.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-07-26T06:22:45.011 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'foo' already exists 2021-07-26T06:22:45.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:22:46.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-07-26T06:22:46.084 INFO:tasks.workunit.client.0.smithi094.stdout:hints = 1 2021-07-26T06:22:46.085 INFO:tasks.workunit.client.0.smithi094.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-07-26T06:22:46.085 INFO:tasks.workunit.client.0.smithi094.stdout:Object prefix: benchmark_data_smithi094_92731 2021-07-26T06:22:46.085 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:22:46.085 INFO:tasks.workunit.client.0.smithi094.stdout: 0 0 0 0 0 0 - 0 2021-07-26T06:22:47.093 INFO:tasks.workunit.client.0.smithi094.stdout: 1 16 48 32 0.124994 0.125 0.32575 0.403766 2021-07-26T06:22:48.082 INFO:tasks.workunit.client.0.smithi094.stdout: 2 16 92 76 0.148426 0.171875 0.347383 0.370595 2021-07-26T06:22:49.083 INFO:tasks.workunit.client.0.smithi094.stdout: 3 16 149 133 0.173162 0.222656 0.286078 0.34245 2021-07-26T06:22:50.074 INFO:tasks.workunit.client.0.smithi094.stdout: 4 16 200 184 0.179671 0.199219 0.280575 0.33261 2021-07-26T06:22:51.075 INFO:tasks.workunit.client.0.smithi094.stdout: 5 16 260 244 0.190605 0.234375 0.285158 0.317846 2021-07-26T06:22:52.079 INFO:tasks.workunit.client.0.smithi094.stdout: 6 16 317 301 0.195941 0.222656 0.202147 0.312479 2021-07-26T06:22:53.073 INFO:tasks.workunit.client.0.smithi094.stdout: 7 16 373 357 0.199196 0.21875 0.273933 0.307265 2021-07-26T06:22:54.079 INFO:tasks.workunit.client.0.smithi094.stdout: 8 16 431 415 0.202614 0.226562 0.251282 0.303845 2021-07-26T06:22:55.074 INFO:tasks.workunit.client.0.smithi094.stdout: 9 16 487 471 0.204402 0.21875 0.243684 0.300024 2021-07-26T06:22:56.085 INFO:tasks.workunit.client.0.smithi094.stdout: 10 16 547 531 0.207397 0.234375 0.315114 0.296074 2021-07-26T06:22:57.084 INFO:tasks.workunit.client.0.smithi094.stdout: 11 16 605 589 0.209136 0.226562 0.318311 0.29576 2021-07-26T06:22:58.103 INFO:tasks.workunit.client.0.smithi094.stdout: 12 16 655 639 0.207981 0.195312 0.308569 0.297329 2021-07-26T06:22:59.075 INFO:tasks.workunit.client.0.smithi094.stdout: 13 16 706 690 0.207305 0.199219 0.251833 0.297737 2021-07-26T06:23:00.075 INFO:tasks.workunit.client.0.smithi094.stdout: 14 16 755 739 0.206167 0.191406 0.230863 0.299135 2021-07-26T06:23:01.079 INFO:tasks.workunit.client.0.smithi094.stdout: 15 16 815 799 0.208046 0.234375 0.293943 0.298021 2021-07-26T06:23:02.078 INFO:tasks.workunit.client.0.smithi094.stdout: 16 16 869 853 0.208225 0.210938 0.284069 0.297806 2021-07-26T06:23:03.076 INFO:tasks.workunit.client.0.smithi094.stdout: 17 16 928 912 0.209532 0.230469 0.280727 0.296179 2021-07-26T06:23:04.076 INFO:tasks.workunit.client.0.smithi094.stdout: 18 16 983 967 0.209825 0.214844 0.254621 0.29521 2021-07-26T06:23:05.086 INFO:tasks.workunit.client.0.smithi094.stdout: 19 16 1041 1025 0.210704 0.226562 0.258777 0.293868 2021-07-26T06:23:06.079 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:23:06.058692+0000 min lat: 0.189648 max lat: 0.636958 avg lat: 0.29411 2021-07-26T06:23:06.079 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:23:06.079 INFO:tasks.workunit.client.0.smithi094.stdout: 20 16 1094 1078 0.210519 0.207031 0.282305 0.29411 2021-07-26T06:23:07.078 INFO:tasks.workunit.client.0.smithi094.stdout: 21 16 1154 1138 0.211653 0.234375 0.289171 0.293762 2021-07-26T06:23:08.079 INFO:tasks.workunit.client.0.smithi094.stdout: 22 16 1204 1188 0.210909 0.195312 0.320744 0.293517 2021-07-26T06:23:09.075 INFO:tasks.workunit.client.0.smithi094.stdout: 23 16 1244 1228 0.208531 0.15625 0.601683 0.294721 2021-07-26T06:23:10.076 INFO:tasks.workunit.client.0.smithi094.stdout: 24 16 1285 1269 0.206514 0.160156 0.290437 0.300466 2021-07-26T06:23:11.095 INFO:tasks.workunit.client.0.smithi094.stdout: 25 16 1347 1331 0.20794 0.242188 0.2715 0.298662 2021-07-26T06:23:12.079 INFO:tasks.workunit.client.0.smithi094.stdout: 26 16 1410 1394 0.209406 0.246094 0.289255 0.297439 2021-07-26T06:23:13.100 INFO:tasks.workunit.client.0.smithi094.stdout: 27 16 1465 1449 0.209606 0.214844 0.319204 0.296357 2021-07-26T06:23:14.083 INFO:tasks.workunit.client.0.smithi094.stdout: 28 16 1520 1504 0.209792 0.214844 0.316804 0.296352 2021-07-26T06:23:15.078 INFO:tasks.workunit.client.0.smithi094.stdout: 29 16 1571 1555 0.209427 0.199219 0.267823 0.296881 2021-07-26T06:23:16.077 INFO:tasks.workunit.client.0.smithi094.stdout: 30 16 1624 1608 0.209346 0.207031 0.316854 0.296968 2021-07-26T06:23:17.077 INFO:tasks.workunit.client.0.smithi094.stdout:Total time run: 30.266 2021-07-26T06:23:17.077 INFO:tasks.workunit.client.0.smithi094.stdout:Total writes made: 1624 2021-07-26T06:23:17.078 INFO:tasks.workunit.client.0.smithi094.stdout:Write size: 4096 2021-07-26T06:23:17.078 INFO:tasks.workunit.client.0.smithi094.stdout:Object size: 4096 2021-07-26T06:23:17.102 INFO:tasks.workunit.client.0.smithi094.stdout:Bandwidth (MB/sec): 0.2096 2021-07-26T06:23:17.103 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Bandwidth: 0.0273457 2021-07-26T06:23:17.103 INFO:tasks.workunit.client.0.smithi094.stdout:Max bandwidth (MB/sec): 0.246094 2021-07-26T06:23:17.103 INFO:tasks.workunit.client.0.smithi094.stdout:Min bandwidth (MB/sec): 0.125 2021-07-26T06:23:17.103 INFO:tasks.workunit.client.0.smithi094.stdout:Average IOPS: 53 2021-07-26T06:23:17.104 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev IOPS: 7.00049 2021-07-26T06:23:17.104 INFO:tasks.workunit.client.0.smithi094.stdout:Max IOPS: 63 2021-07-26T06:23:17.104 INFO:tasks.workunit.client.0.smithi094.stdout:Min IOPS: 32 2021-07-26T06:23:17.104 INFO:tasks.workunit.client.0.smithi094.stdout:Average Latency(s): 0.296923 2021-07-26T06:23:17.104 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Latency(s): 0.063545 2021-07-26T06:23:17.105 INFO:tasks.workunit.client.0.smithi094.stdout:Max latency(s): 0.869564 2021-07-26T06:23:17.105 INFO:tasks.workunit.client.0.smithi094.stdout:Min latency(s): 0.184221 2021-07-26T06:23:17.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2021-07-26T06:23:17.212 INFO:tasks.workunit.client.0.smithi094.stdout:after bench 2021-07-26T06:23:17.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 84977 2021-07-26T06:23:17.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2021-07-26T06:23:18.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 84977 2021-07-26T06:23:18.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (84977) - No such process 2021-07-26T06:23:18.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2021-07-26T06:23:18.657 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:23:18.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 87218 2021-07-26T06:23:18.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2021-07-26T06:23:19.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 87218 2021-07-26T06:23:19.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (87218) - No such process 2021-07-26T06:23:19.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2021-07-26T06:23:20.126 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T06:23:20.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 89459 2021-07-26T06:23:20.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2021-07-26T06:23:21.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 89459 2021-07-26T06:23:21.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (89459) - No such process 2021-07-26T06:23:21.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2021-07-26T06:23:21.541 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already down. 2021-07-26T06:23:21.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 91700 2021-07-26T06:23:21.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2021-07-26T06:23:22.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 91700 2021-07-26T06:23:22.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (91700) - No such process 2021-07-26T06:23:22.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2021-07-26T06:23:23.021 INFO:tasks.workunit.client.0.smithi094.stderr:osd.3 is already down. 2021-07-26T06:23:23.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:23:24.859 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:23:24.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:23:26.661 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:23:26.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-07-26T06:23:28.466 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:23:28.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:23:30.278 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:23:30.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2021-07-26T06:23:30.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2021-07-26T06:23:30.326 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:23:31.218 INFO:tasks.workunit.client.0.smithi094.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-07-26T06:23:31.218 INFO:tasks.workunit.client.0.smithi094.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-07-26T06:23:31.221 INFO:tasks.workunit.client.0.smithi094.stdout:2 : device size 0x100000000 : using 0x804ea000(2.0 GiB) 2021-07-26T06:23:31.221 INFO:tasks.workunit.client.0.smithi094.stdout:Expanding DB/WAL... 2021-07-26T06:23:31.221 INFO:tasks.workunit.client.0.smithi094.stdout:0 : expanding to 0x536870912 2021-07-26T06:23:31.231 INFO:tasks.workunit.client.0.smithi094.stdout:0 : size label updated to 536870912 2021-07-26T06:23:31.232 INFO:tasks.workunit.client.0.smithi094.stdout:1 : expanding to 0x1073741824 2021-07-26T06:23:31.252 INFO:tasks.workunit.client.0.smithi094.stdout:1 : size label updated to 1073741824 2021-07-26T06:23:31.252 INFO:tasks.workunit.client.0.smithi094.stdout:2 : expanding from 0x80000000 to 0x100000000 2021-07-26T06:23:31.503 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:31.465+0000 7ffb4ad22240 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-07-26T06:23:31.503 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:31.465+0000 7ffb4ad22240 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-07-26T06:23:37.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2021-07-26T06:23:37.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2021-07-26T06:23:37.192 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:23:38.062 INFO:tasks.workunit.client.0.smithi094.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-07-26T06:23:38.062 INFO:tasks.workunit.client.0.smithi094.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-07-26T06:23:38.063 INFO:tasks.workunit.client.0.smithi094.stdout:2 : device size 0x101000000 : using 0x814f8000(2.0 GiB) 2021-07-26T06:23:38.063 INFO:tasks.workunit.client.0.smithi094.stdout:Expanding DB/WAL... 2021-07-26T06:23:38.063 INFO:tasks.workunit.client.0.smithi094.stdout:0 : expanding to 0x536870912 2021-07-26T06:23:38.109 INFO:tasks.workunit.client.0.smithi094.stdout:0 : size label updated to 536870912 2021-07-26T06:23:38.109 INFO:tasks.workunit.client.0.smithi094.stdout:1 : expanding to 0x1073741824 2021-07-26T06:23:38.127 INFO:tasks.workunit.client.0.smithi094.stdout:1 : size label updated to 1073741824 2021-07-26T06:23:38.128 INFO:tasks.workunit.client.0.smithi094.stdout:2 : expanding from 0x80000000 to 0x101000000 2021-07-26T06:23:38.347 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:38.324+0000 7f514f600240 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-07-26T06:23:38.347 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:38.324+0000 7f514f600240 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-07-26T06:23:43.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2021-07-26T06:23:43.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2021-07-26T06:23:43.488 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:23:44.349 INFO:tasks.workunit.client.0.smithi094.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-07-26T06:23:44.349 INFO:tasks.workunit.client.0.smithi094.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-07-26T06:23:44.349 INFO:tasks.workunit.client.0.smithi094.stdout:2 : device size 0x100020000 : using 0x80552000(2.0 GiB) 2021-07-26T06:23:44.350 INFO:tasks.workunit.client.0.smithi094.stdout:Expanding DB/WAL... 2021-07-26T06:23:44.350 INFO:tasks.workunit.client.0.smithi094.stdout:0 : expanding to 0x536870912 2021-07-26T06:23:44.387 INFO:tasks.workunit.client.0.smithi094.stdout:0 : size label updated to 536870912 2021-07-26T06:23:44.387 INFO:tasks.workunit.client.0.smithi094.stdout:1 : expanding to 0x1073741824 2021-07-26T06:23:44.407 INFO:tasks.workunit.client.0.smithi094.stdout:1 : size label updated to 1073741824 2021-07-26T06:23:44.408 INFO:tasks.workunit.client.0.smithi094.stdout:2 : expanding from 0x80000000 to 0x100020000 2021-07-26T06:23:44.620 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:44.596+0000 7fdc47a56240 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-07-26T06:23:44.620 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:44.596+0000 7fdc47a56240 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-07-26T06:23:50.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2021-07-26T06:23:50.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2021-07-26T06:23:50.117 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:23:50.982 INFO:tasks.workunit.client.0.smithi094.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-07-26T06:23:50.983 INFO:tasks.workunit.client.0.smithi094.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-07-26T06:23:50.983 INFO:tasks.workunit.client.0.smithi094.stdout:2 : device size 0xfff00000 : using 0x803ef000(2.0 GiB) 2021-07-26T06:23:50.983 INFO:tasks.workunit.client.0.smithi094.stdout:Expanding DB/WAL... 2021-07-26T06:23:50.984 INFO:tasks.workunit.client.0.smithi094.stdout:0 : expanding to 0x536870912 2021-07-26T06:23:51.018 INFO:tasks.workunit.client.0.smithi094.stdout:0 : size label updated to 536870912 2021-07-26T06:23:51.018 INFO:tasks.workunit.client.0.smithi094.stdout:1 : expanding to 0x1073741824 2021-07-26T06:23:51.044 INFO:tasks.workunit.client.0.smithi094.stdout:1 : size label updated to 1073741824 2021-07-26T06:23:51.045 INFO:tasks.workunit.client.0.smithi094.stdout:2 : expanding from 0x80000000 to 0xfff00000 2021-07-26T06:23:51.260 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:51.240+0000 7fdd30af2240 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-07-26T06:23:51.261 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:23:51.240+0000 7fdd30af2240 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-07-26T06:23:56.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:23:58.626 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:23:58.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:24:00.441 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:00.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-07-26T06:24:02.240 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:02.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:24:04.062 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:04.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2021-07-26T06:24:04.149 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:24:05.887 INFO:tasks.workunit.client.0.smithi094.stdout:0 : device size 0x1ffff000 : using 0x500000(5 MiB) 2021-07-26T06:24:05.887 INFO:tasks.workunit.client.0.smithi094.stdout:1 : device size 0x3fffe000 : using 0x1000000(16 MiB) 2021-07-26T06:24:05.888 INFO:tasks.workunit.client.0.smithi094.stdout:2 : device size 0x100000000 : using 0x4ea000(4.9 MiB) 2021-07-26T06:24:05.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2021-07-26T06:24:05.937 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:24:07.701 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-07-26T06:24:07.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:24:09.022 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:09.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-07-26T06:24:09.055 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:24:10.858 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-07-26T06:24:10.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:24:12.163 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:12.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2021-07-26T06:24:12.203 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:24:13.735 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-07-26T06:24:13.736 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-07-26T06:24:13.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-07-26T06:24:14.547 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:14.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2021-07-26T06:24:14.576 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:24:14.580 INFO:tasks.workunit.client.0.smithi094.stderr:Migrate to WAL device isn't supported. 2021-07-26T06:24:14.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2021-07-26T06:24:14.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:24:16.391 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:16.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-07-26T06:24:16.428 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:24:18.203 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-07-26T06:24:18.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:24:19.514 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:19.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2021-07-26T06:24:19.535 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:24:20.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:24:22.127 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:24:22.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-07-26T06:24:22.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:24:22.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:24:22.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:24:22.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:24:22.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-07-26T06:24:22.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:24:22.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:24:22.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:24:22.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:24:22.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-07-26T06:24:22.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-07-26T06:24:22.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:24:22.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:24:22.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:24:22.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:24:22.161 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:24:22.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:24:22.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:24:22.162 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:24:22.162 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:22.164 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:24:22.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:22.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:22.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:24:22.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:24:22.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:24:22.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:24:22.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:24:22.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:22.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:24:22.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:24:22.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-07-26T06:24:22.167 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:24:22.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:24:22.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:22.169 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-07-26T06:24:22.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:24:22.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:24:22.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:24:22.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:24:22.171 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:22.146+0000 7f68a9af6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:22.172 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:22.150+0000 7f68a9af6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:22.185 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:22.150+0000 7f68a9af6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:22.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:24:22.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:24:22.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:24:22.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:24:22.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:24:22.521 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:24:22.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:22.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:24:22.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:22.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:24:22.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:22.973 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:22.950+0000 7f68a9af6f40 -1 Falling back to public interface 2021-07-26T06:24:23.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:23.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:23.913 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:24:23.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:24:23.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:23.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:24:24.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:25.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:25.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:25.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:24:25.252 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:24:25.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:25.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:24:25.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:26.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:26.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:26.730 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:24:26.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:24:26.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:26.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:24:27.031 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:27.010+0000 7f68a9af6f40 -1 osd.0 26 log_to_monitors {default=true} 2021-07-26T06:24:27.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:28.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:28.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:28.156 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:24:28.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:24:28.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:28.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:24:28.460 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 35 up_thru 35 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/105062,v1:127.0.0.1:6803/105062] [v2:127.0.0.1:6804/105062,v1:127.0.0.1:6805/105062] exists,up 48983ae5-ea84-4702-9a7d-c6e04ee74555 2021-07-26T06:24:28.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:24:28.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:24:28.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:24:28.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-07-26T06:24:28.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=105062 2021-07-26T06:24:28.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-07-26T06:24:28.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:24:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:24:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:24:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:24:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-07-26T06:24:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:24:28.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:24:28.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:24:28.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:24:28.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-07-26T06:24:28.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-07-26T06:24:28.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:24:28.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:24:28.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:24:28.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:24:28.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:24:28.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:24:28.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:24:28.587 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:24:28.587 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:28.588 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:24:28.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:28.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:28.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:24:28.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:24:28.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:24:28.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:24:28.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:24:28.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:28.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:24:28.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:24:28.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-07-26T06:24:28.599 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:24:28.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:24:28.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:28.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-07-26T06:24:28.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:24:28.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:24:28.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:24:28.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:24:28.602 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:28.474+0000 7f78380d8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:28.602 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:28.494+0000 7f78380d8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:28.602 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:28.494+0000 7f78380d8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:28.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:24:28.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:24:28.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:24:28.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:24:28.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:24:28.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:28.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:24:28.908 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:24:28.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:28.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:24:29.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:29.334 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:29.310+0000 7f78380d8f40 -1 Falling back to public interface 2021-07-26T06:24:30.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:30.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:30.280 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:24:30.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:24:30.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:30.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:24:30.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:31.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:31.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:31.640 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:24:31.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:24:31.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:31.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:24:32.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:33.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:33.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:33.012 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:24:33.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:24:33.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:33.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:24:33.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:33.547 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:33.514+0000 7f78380d8f40 -1 osd.1 28 log_to_monitors {default=true} 2021-07-26T06:24:34.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:34.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:34.399 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:24:34.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:24:34.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:34.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:24:34.949 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/106044,v1:127.0.0.1:6811/106044] [v2:127.0.0.1:6812/106044,v1:127.0.0.1:6813/106044] exists,up 817d3194-fc16-4662-9b83-ea4d7aabf413 2021-07-26T06:24:34.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:24:34.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:24:34.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:24:34.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-07-26T06:24:34.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=106044 2021-07-26T06:24:34.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-07-26T06:24:34.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:24:34.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:24:34.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:24:34.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:24:34.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-07-26T06:24:34.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:24:34.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:24:34.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:24:34.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:24:34.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-07-26T06:24:34.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-07-26T06:24:34.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:24:34.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:24:34.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:24:34.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:24:34.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:24:34.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:24:34.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:24:34.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:24:34.990 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:34.990 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:24:34.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:35.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:35.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:24:35.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:24:35.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:24:35.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:24:35.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:24:35.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:35.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:24:35.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:24:35.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-07-26T06:24:35.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:24:35.017 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:24:35.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:35.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-07-26T06:24:35.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:24:35.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:24:35.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:24:35.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:24:35.030 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:34.958+0000 7f21d3acaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:35.083 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:35.058+0000 7f21d3acaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:35.084 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:35.058+0000 7f21d3acaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:35.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:24:35.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:24:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:24:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:24:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:24:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:24:35.424 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:24:35.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:35.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:24:35.645 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:35.610+0000 7f21d3acaf40 -1 Falling back to public interface 2021-07-26T06:24:35.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:36.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:36.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:36.776 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:24:36.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:24:36.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:36.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:24:37.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:38.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:38.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:38.194 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:24:38.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:24:38.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:38.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:24:38.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:39.432 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:39.405+0000 7f21d3acaf40 -1 osd.2 30 log_to_monitors {default=true} 2021-07-26T06:24:39.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:39.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:39.513 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:24:39.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:24:39.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:39.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:24:40.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:41.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:41.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:41.116 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:24:41.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:24:41.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:41.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:24:41.866 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 41 up_thru 41 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/107041,v1:127.0.0.1:6819/107041] [v2:127.0.0.1:6820/107041,v1:127.0.0.1:6821/107041] exists,up 47d572b5-8285-4254-ab35-52b7fa3c0ddf 2021-07-26T06:24:41.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:24:41.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:24:41.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:24:41.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-07-26T06:24:41.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=107041 2021-07-26T06:24:41.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-07-26T06:24:41.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:24:41.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:24:41.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:24:41.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:24:41.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-07-26T06:24:41.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:24:41.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:24:41.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:24:41.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:24:41.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-07-26T06:24:41.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-07-26T06:24:41.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:24:41.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:24:41.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:24:41.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:24:41.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:24:41.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:24:41.888 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:24:41.888 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:24:41.888 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:24:41.888 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:24:41.889 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:41.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:24:41.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:24:41.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:24:41.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:24:41.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:24:41.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:24:41.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:41.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:24:41.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:24:41.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-07-26T06:24:41.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:24:41.899 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T06:24:41.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:24:41.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-07-26T06:24:41.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:24:41.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:24:41.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:24:41.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:24:41.902 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:41.877+0000 7f21af33ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:41.916 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:41.881+0000 7f21af33ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:41.917 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:41.885+0000 7f21af33ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:24:42.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:24:42.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:24:42.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:24:42.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:24:42.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:24:42.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:42.246 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:24:42.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:24:42.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:42.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:24:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:42.710 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:42.685+0000 7f21af33ef40 -1 Falling back to public interface 2021-07-26T06:24:43.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:43.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:43.640 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:24:43.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:24:43.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:43.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:24:43.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:44.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:44.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:44.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:24:44.992 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:24:44.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:44.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:24:45.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:46.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:46.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:46.356 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:24:46.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:24:46.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:46.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:24:46.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:24:47.105 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:24:47.081+0000 7f21af33ef40 -1 osd.3 32 log_to_monitors {default=true} 2021-07-26T06:24:47.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:24:47.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:24:47.749 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:24:47.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:24:47.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:24:47.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:24:48.399 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 46 up_thru 31 down_at 33 last_clean_interval [20,32) [v2:127.0.0.1:6826/107998,v1:127.0.0.1:6827/107998] [v2:127.0.0.1:6828/107998,v1:127.0.0.1:6829/107998] exists,up 93101700-5763-487e-b234-f4cc2d81e938 2021-07-26T06:24:48.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:24:48.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:24:48.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:24:48.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-07-26T06:24:48.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=107998 2021-07-26T06:24:48.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2021-07-26T06:24:48.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:24:48.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:24:48.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:24:48.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:24:48.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:24:48.405 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:24:48.406 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:24:48.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:24:48.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:24:48.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:24:48.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:24:48.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:24:48.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:24:48.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:24:48.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:24:49.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:24:49.099 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:24:49.099 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:24:49.099 INFO:tasks.workunit.client.0.smithi094.stderr:3' 2021-07-26T06:24:49.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:24:49.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:24:49.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:24:49.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=150323855367 2021-07-26T06:24:49.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 150323855367 2021-07-26T06:24:49.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855367' 2021-07-26T06:24:49.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:24:49.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:24:49.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=163208757254 2021-07-26T06:24:49.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 163208757254 2021-07-26T06:24:49.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855367 1-163208757254' 2021-07-26T06:24:49.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:24:49.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:24:49.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=176093659140 2021-07-26T06:24:49.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 176093659140 2021-07-26T06:24:49.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855367 1-163208757254 2-176093659140' 2021-07-26T06:24:49.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:24:49.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:24:49.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=197568495619 2021-07-26T06:24:49.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 197568495619 2021-07-26T06:24:49.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-150323855367 1-163208757254 2-176093659140 3-197568495619' 2021-07-26T06:24:49.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:24:49.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-150323855367 2021-07-26T06:24:49.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:24:49.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:24:49.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-150323855367 2021-07-26T06:24:49.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:24:49.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=150323855367 2021-07-26T06:24:49.719 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 150323855367 2021-07-26T06:24:49.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 150323855367' 2021-07-26T06:24:49.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:24:50.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 150323855367 -lt 150323855367 2021-07-26T06:24:50.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:24:50.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-163208757254 2021-07-26T06:24:50.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:24:50.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:24:50.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-163208757254 2021-07-26T06:24:50.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:24:50.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=163208757254 2021-07-26T06:24:50.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 163208757254' 2021-07-26T06:24:50.105 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 163208757254 2021-07-26T06:24:50.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:24:50.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 163208757254 -lt 163208757254 2021-07-26T06:24:50.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:24:50.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:24:50.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-176093659140 2021-07-26T06:24:50.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:24:50.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:24:50.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-176093659140 2021-07-26T06:24:50.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=176093659140 2021-07-26T06:24:50.507 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 176093659140 2021-07-26T06:24:50.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 176093659140' 2021-07-26T06:24:50.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:24:50.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 176093659138 -lt 176093659140 2021-07-26T06:24:50.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:24:51.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:24:51.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:24:52.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 176093659141 -lt 176093659140 2021-07-26T06:24:52.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:24:52.256 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-197568495619 2021-07-26T06:24:52.256 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:24:52.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:24:52.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-197568495619 2021-07-26T06:24:52.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:24:52.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=197568495619 2021-07-26T06:24:52.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 197568495619' 2021-07-26T06:24:52.273 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 197568495619 2021-07-26T06:24:52.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:24:52.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 197568495619 -lt 197568495619 2021-07-26T06:24:52.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:24:52.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:24:52.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:24:53.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 16 == 0 2021-07-26T06:24:53.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:24:53.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:24:53.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:24:53.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:24:53.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:24:53.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:24:53.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:24:53.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=16 2021-07-26T06:24:53.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:24:53.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:24:53.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:24:53.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 16 = 16 2021-07-26T06:24:53.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:24:53.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:24:53.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-07-26T06:24:53.986 INFO:tasks.workunit.client.0.smithi094.stdout:hints = 1 2021-07-26T06:24:53.986 INFO:tasks.workunit.client.0.smithi094.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-07-26T06:24:53.986 INFO:tasks.workunit.client.0.smithi094.stdout:Object prefix: benchmark_data_smithi094_109412 2021-07-26T06:24:53.987 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:24:53.987 INFO:tasks.workunit.client.0.smithi094.stdout: 0 0 0 0 0 0 - 0 2021-07-26T06:24:54.983 INFO:tasks.workunit.client.0.smithi094.stdout: 1 16 35 19 0.0742109 0.0742188 0.222499 0.287223 2021-07-26T06:24:55.996 INFO:tasks.workunit.client.0.smithi094.stdout: 2 16 40 24 0.0468682 0.0195312 0.117153 0.261402 2021-07-26T06:24:56.983 INFO:tasks.workunit.client.0.smithi094.stdout: 3 16 40 24 0.0312454 0 - 0.261402 2021-07-26T06:24:57.983 INFO:tasks.workunit.client.0.smithi094.stdout: 4 16 40 24 0.0234342 0 - 0.261402 2021-07-26T06:24:58.987 INFO:tasks.workunit.client.0.smithi094.stdout: 5 16 71 55 0.0429628 0.0403646 0.300414 1.07296 2021-07-26T06:24:59.984 INFO:tasks.workunit.client.0.smithi094.stdout: 6 16 120 104 0.0676985 0.191406 0.309939 0.895192 2021-07-26T06:25:00.984 INFO:tasks.workunit.client.0.smithi094.stdout: 7 16 183 167 0.0931782 0.246094 0.315222 0.664538 2021-07-26T06:25:01.990 INFO:tasks.workunit.client.0.smithi094.stdout: 8 16 231 215 0.104966 0.1875 0.401038 0.592715 2021-07-26T06:25:02.987 INFO:tasks.workunit.client.0.smithi094.stdout: 9 16 265 249 0.108058 0.132812 0.311919 0.560698 2021-07-26T06:25:03.990 INFO:tasks.workunit.client.0.smithi094.stdout: 10 16 327 311 0.121467 0.242188 0.341718 0.507898 2021-07-26T06:25:05.017 INFO:tasks.workunit.client.0.smithi094.stdout: 11 16 377 361 0.128177 0.195312 0.287136 0.481483 2021-07-26T06:25:06.004 INFO:tasks.workunit.client.0.smithi094.stdout: 12 16 425 409 0.133119 0.1875 0.311193 0.459118 2021-07-26T06:25:06.989 INFO:tasks.workunit.client.0.smithi094.stdout: 13 16 487 471 0.141506 0.242188 0.288178 0.438216 2021-07-26T06:25:07.996 INFO:tasks.workunit.client.0.smithi094.stdout: 14 16 535 519 0.14479 0.1875 0.386825 0.429198 2021-07-26T06:25:08.986 INFO:tasks.workunit.client.0.smithi094.stdout: 15 16 583 567 0.147635 0.1875 0.289236 0.418216 2021-07-26T06:25:09.987 INFO:tasks.workunit.client.0.smithi094.stdout: 16 16 634 618 0.150857 0.199219 0.316194 0.407727 2021-07-26T06:25:10.988 INFO:tasks.workunit.client.0.smithi094.stdout: 17 16 695 679 0.155998 0.238281 0.334777 0.398896 2021-07-26T06:25:11.986 INFO:tasks.workunit.client.0.smithi094.stdout: 18 16 743 727 0.157747 0.1875 0.321647 0.393772 2021-07-26T06:25:12.986 INFO:tasks.workunit.client.0.smithi094.stdout: 19 16 778 762 0.156639 0.136719 0.551531 0.396262 2021-07-26T06:25:13.991 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:25:13.966043+0000 min lat: 0.104791 max lat: 5.19887 avg lat: 0.391045 2021-07-26T06:25:13.992 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:25:13.992 INFO:tasks.workunit.client.0.smithi094.stdout: 20 16 826 810 0.158181 0.1875 0.318528 0.391045 2021-07-26T06:25:14.989 INFO:tasks.workunit.client.0.smithi094.stdout: 21 16 887 871 0.161993 0.238281 0.296839 0.384462 2021-07-26T06:25:15.986 INFO:tasks.workunit.client.0.smithi094.stdout: 22 16 938 922 0.163684 0.199219 0.304886 0.378769 2021-07-26T06:25:16.987 INFO:tasks.workunit.client.0.smithi094.stdout: 23 16 999 983 0.166926 0.238281 0.292809 0.373341 2021-07-26T06:25:17.991 INFO:tasks.workunit.client.0.smithi094.stdout: 24 16 1047 1031 0.167782 0.1875 0.317729 0.370221 2021-07-26T06:25:18.989 INFO:tasks.workunit.client.0.smithi094.stdout: 25 16 1095 1079 0.168569 0.1875 0.308763 0.368465 2021-07-26T06:25:19.988 INFO:tasks.workunit.client.0.smithi094.stdout: 26 16 1143 1127 0.169296 0.1875 0.319818 0.366545 2021-07-26T06:25:20.993 INFO:tasks.workunit.client.0.smithi094.stdout: 27 16 1194 1178 0.170403 0.199219 0.247889 0.363561 2021-07-26T06:25:21.988 INFO:tasks.workunit.client.0.smithi094.stdout: 28 16 1242 1226 0.171013 0.1875 0.311501 0.362507 2021-07-26T06:25:22.990 INFO:tasks.workunit.client.0.smithi094.stdout: 29 16 1303 1287 0.173331 0.238281 0.330595 0.359808 2021-07-26T06:25:23.988 INFO:tasks.workunit.client.0.smithi094.stdout: 30 16 1351 1335 0.173802 0.1875 0.336756 0.357987 2021-07-26T06:25:24.988 INFO:tasks.workunit.client.0.smithi094.stdout:Total time run: 30.2079 2021-07-26T06:25:24.989 INFO:tasks.workunit.client.0.smithi094.stdout:Total writes made: 1351 2021-07-26T06:25:24.989 INFO:tasks.workunit.client.0.smithi094.stdout:Write size: 4096 2021-07-26T06:25:24.989 INFO:tasks.workunit.client.0.smithi094.stdout:Object size: 4096 2021-07-26T06:25:24.989 INFO:tasks.workunit.client.0.smithi094.stdout:Bandwidth (MB/sec): 0.174701 2021-07-26T06:25:24.989 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Bandwidth: 0.0721021 2021-07-26T06:25:24.990 INFO:tasks.workunit.client.0.smithi094.stdout:Max bandwidth (MB/sec): 0.246094 2021-07-26T06:25:24.990 INFO:tasks.workunit.client.0.smithi094.stdout:Min bandwidth (MB/sec): 0 2021-07-26T06:25:24.990 INFO:tasks.workunit.client.0.smithi094.stdout:Average IOPS: 44 2021-07-26T06:25:24.990 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev IOPS: 18.4791 2021-07-26T06:25:24.991 INFO:tasks.workunit.client.0.smithi094.stdout:Max IOPS: 63 2021-07-26T06:25:24.991 INFO:tasks.workunit.client.0.smithi094.stdout:Min IOPS: 0 2021-07-26T06:25:24.991 INFO:tasks.workunit.client.0.smithi094.stdout:Average Latency(s): 0.357441 2021-07-26T06:25:24.991 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Latency(s): 0.421382 2021-07-26T06:25:24.991 INFO:tasks.workunit.client.0.smithi094.stdout:Max latency(s): 5.19887 2021-07-26T06:25:24.992 INFO:tasks.workunit.client.0.smithi094.stdout:Min latency(s): 0.104791 2021-07-26T06:25:25.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 105062 2021-07-26T06:25:25.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2021-07-26T06:25:26.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 105062 2021-07-26T06:25:26.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (105062) - No such process 2021-07-26T06:25:26.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2021-07-26T06:25:26.649 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:25:26.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 106044 2021-07-26T06:25:26.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2021-07-26T06:25:27.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 106044 2021-07-26T06:25:27.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (106044) - No such process 2021-07-26T06:25:27.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2021-07-26T06:25:28.050 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T06:25:28.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 107041 2021-07-26T06:25:28.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2021-07-26T06:25:29.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 107041 2021-07-26T06:25:29.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (107041) - No such process 2021-07-26T06:25:29.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2021-07-26T06:25:29.428 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already down. 2021-07-26T06:25:29.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 107998 2021-07-26T06:25:29.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2021-07-26T06:25:30.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 107998 2021-07-26T06:25:30.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (107998) - No such process 2021-07-26T06:25:30.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2021-07-26T06:25:30.999 INFO:tasks.workunit.client.0.smithi094.stderr:osd.3 is already down. 2021-07-26T06:25:31.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:25:32.314 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:25:32.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2021-07-26T06:25:32.607 INFO:tasks.workunit.client.0.smithi094.stderr:512+0 records in 2021-07-26T06:25:32.608 INFO:tasks.workunit.client.0.smithi094.stderr:512+0 records out 2021-07-26T06:25:32.608 INFO:tasks.workunit.client.0.smithi094.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.2876 s, 1.9 GB/s 2021-07-26T06:25:32.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2021-07-26T06:25:32.625 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:25:37.686 INFO:tasks.workunit.client.0.smithi094.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2021-07-26T06:25:37.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:25:39.512 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:25:39.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:25:40.821 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:25:40.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2021-07-26T06:25:41.355 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records in 2021-07-26T06:25:41.356 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records out 2021-07-26T06:25:41.357 INFO:tasks.workunit.client.0.smithi094.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.526741 s, 2.0 GB/s 2021-07-26T06:25:41.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2021-07-26T06:25:41.375 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:25:50.171 INFO:tasks.workunit.client.0.smithi094.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2021-07-26T06:25:50.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2021-07-26T06:25:50.192 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:25:51.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:25:53.785 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:25:53.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-07-26T06:25:54.601 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:25:54.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2021-07-26T06:25:54.623 INFO:tasks.workunit.client.0.smithi094.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-07-26T06:25:54.623 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:25:55.410 INFO:tasks.workunit.client.0.smithi094.stdout:DB device added 2021-07-26T06:25:55.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2021-07-26T06:25:55.433 INFO:tasks.workunit.client.0.smithi094.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-07-26T06:25:55.433 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:25:56.726 INFO:tasks.workunit.client.0.smithi094.stdout:WAL device added 2021-07-26T06:25:56.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2021-07-26T06:25:56.747 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:25:58.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-07-26T06:26:00.599 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:26:00.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:26:01.918 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:26:01.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2021-07-26T06:26:01.943 INFO:tasks.workunit.client.0.smithi094.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-07-26T06:26:01.943 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:26:03.230 INFO:tasks.workunit.client.0.smithi094.stdout:WAL device added 2021-07-26T06:26:03.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-07-26T06:26:03.255 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:26:05.046 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2021-07-26T06:26:05.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:26:06.366 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:26:06.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-07-26T06:26:06.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:26:06.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:26:06.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:26:06.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:26:06.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-07-26T06:26:06.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:26:06.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:26:06.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:26:06.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:26:06.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-07-26T06:26:06.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-07-26T06:26:06.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:26:06.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:26:06.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:26:06.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:26:06.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:26:06.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:26:06.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:26:06.379 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:26:06.380 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:06.380 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:26:06.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:06.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:06.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:26:06.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:26:06.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:26:06.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:26:06.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:26:06.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:06.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:26:06.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:26:06.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-07-26T06:26:06.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:26:06.384 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:26:06.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:06.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-07-26T06:26:06.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:26:06.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:26:06.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:26:06.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:26:06.409 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:06.381+0000 7fee10225f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:06.410 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:06.385+0000 7fee10225f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:06.417 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:06.385+0000 7fee10225f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:06.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:26:06.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:26:06.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:26:06.736 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:26:06.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:26:06.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:26:06.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:06.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:26:06.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:06.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:26:07.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:07.452 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:07.425+0000 7fee10225f40 -1 Falling back to public interface 2021-07-26T06:26:08.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:08.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:08.128 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:26:08.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:26:08.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:08.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:26:08.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:09.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:09.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:09.521 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:26:09.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:26:09.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:09.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:26:09.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:10.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:10.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:10.943 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:26:10.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:26:10.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:10.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:26:11.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:12.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:12.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:12.291 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:26:12.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:26:12.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:12.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:26:12.419 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:12.389+0000 7fee10225f40 -1 osd.0 47 log_to_monitors {default=true} 2021-07-26T06:26:12.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:13.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:13.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:13.736 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:26:13.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:26:13.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:13.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:26:14.114 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 57 up_thru 57 down_at 48 last_clean_interval [35,47) [v2:127.0.0.1:6802/115675,v1:127.0.0.1:6803/115675] [v2:127.0.0.1:6804/115675,v1:127.0.0.1:6805/115675] exists,up 48983ae5-ea84-4702-9a7d-c6e04ee74555 2021-07-26T06:26:14.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:26:14.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:26:14.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:26:14.125 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-07-26T06:26:14.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=115675 2021-07-26T06:26:14.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-07-26T06:26:14.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:26:14.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:26:14.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:26:14.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:26:14.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-07-26T06:26:14.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:26:14.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:26:14.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:26:14.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:26:14.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-07-26T06:26:14.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-07-26T06:26:14.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:26:14.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:26:14.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:26:14.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:26:14.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:26:14.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:26:14.163 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:26:14.163 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:26:14.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:14.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:26:14.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:14.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:14.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:26:14.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:26:14.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:26:14.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:26:14.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:26:14.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:14.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:26:14.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:26:14.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-07-26T06:26:14.184 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:26:14.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:26:14.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:14.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-07-26T06:26:14.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:26:14.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:26:14.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:26:14.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:26:14.187 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:14.125+0000 7fe2f1dd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:14.208 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:14.165+0000 7fe2f1dd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:14.208 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:14.165+0000 7fe2f1dd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:14.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:26:14.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:26:14.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:26:14.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:26:14.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:26:14.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:14.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:26:14.503 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:26:14.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:14.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:26:14.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:15.245 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:15.213+0000 7fe2f1dd0f40 -1 Falling back to public interface 2021-07-26T06:26:15.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:15.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:15.937 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:26:15.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:26:15.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:15.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:26:16.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:17.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:17.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:17.310 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:26:17.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:26:17.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:17.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:26:17.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:18.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:18.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:18.758 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:26:18.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:26:18.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:18.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:26:19.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:20.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:20.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:20.117 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:26:20.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:26:20.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:20.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:26:20.234 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:20.205+0000 7fe2f1dd0f40 -1 osd.1 50 log_to_monitors {default=true} 2021-07-26T06:26:20.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:21.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:21.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:21.500 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:26:21.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:26:21.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:21.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:26:21.857 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 51 last_clean_interval [38,50) [v2:127.0.0.1:6810/116694,v1:127.0.0.1:6811/116694] [v2:127.0.0.1:6812/116694,v1:127.0.0.1:6813/116694] exists,up 817d3194-fc16-4662-9b83-ea4d7aabf413 2021-07-26T06:26:21.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:26:21.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:26:21.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:26:21.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-07-26T06:26:21.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=116694 2021-07-26T06:26:21.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-07-26T06:26:21.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:26:21.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:26:21.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:26:21.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:26:21.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-07-26T06:26:21.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:26:21.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:26:21.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:26:21.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:26:21.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-07-26T06:26:21.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-07-26T06:26:21.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:26:21.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:26:21.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:26:21.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:26:21.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:26:21.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:26:21.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:26:21.866 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:26:21.866 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:21.866 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:26:21.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:21.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:21.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:26:21.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:26:21.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:26:21.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:26:21.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:26:21.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:21.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:26:21.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:26:21.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-07-26T06:26:21.871 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:26:21.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:26:21.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:21.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-07-26T06:26:21.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:26:21.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:26:21.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:26:21.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:26:21.902 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:21.876+0000 7ff665fdef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:21.958 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:21.928+0000 7ff665fdef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:21.960 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:21.932+0000 7ff665fdef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:22.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:26:22.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:26:22.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:26:22.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:26:22.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:26:22.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:22.306 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:26:22.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:26:22.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:22.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:26:22.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:22.771 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:22.736+0000 7ff665fdef40 -1 Falling back to public interface 2021-07-26T06:26:23.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:23.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:23.639 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:26:23.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:26:23.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:23.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:26:23.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:24.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:24.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:24.999 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:26:25.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:26:25.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:25.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:26:25.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:26.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:26.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:26.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:26:26.380 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:26:26.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:26.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:26:26.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:27.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:27.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:27.730 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:26:27.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:26:27.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:27.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:26:28.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:28.536+0000 7ff665fdef40 -1 osd.2 52 log_to_monitors {default=true} 2021-07-26T06:26:29.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:29.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:29.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:26:29.179 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:26:29.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:29.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:26:29.721 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 63 up_thru 63 down_at 53 last_clean_interval [41,52) [v2:127.0.0.1:6818/117730,v1:127.0.0.1:6819/117730] [v2:127.0.0.1:6820/117730,v1:127.0.0.1:6821/117730] exists,up 47d572b5-8285-4254-ab35-52b7fa3c0ddf 2021-07-26T06:26:29.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:26:29.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:26:29.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:26:29.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-07-26T06:26:29.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=117730 2021-07-26T06:26:29.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-07-26T06:26:29.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:26:29.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:26:29.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:26:29.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:26:29.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-07-26T06:26:29.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:26:29.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:26:29.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:26:29.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:26:29.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-07-26T06:26:29.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-07-26T06:26:29.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:26:29.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:26:29.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:26:29.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:26:29.757 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:26:29.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:26:29.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:26:29.759 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:26:29.759 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:26:29.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:26:29.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:29.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:26:29.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:26:29.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:26:29.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:26:29.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:26:29.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:26:29.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:29.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:26:29.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:26:29.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-07-26T06:26:29.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:26:29.811 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T06:26:29.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:26:29.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-07-26T06:26:29.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:26:29.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:26:29.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:26:29.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:26:29.813 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:29.732+0000 7f419c84df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:29.975 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:29.944+0000 7f419c84df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:29.978 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:29.948+0000 7f419c84df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:26:30.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:26:30.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:26:30.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:26:30.388 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:26:30.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:26:30.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:26:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:26:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:26:30.766 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:30.736+0000 7f419c84df40 -1 Falling back to public interface 2021-07-26T06:26:30.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:31.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:31.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:31.780 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:26:31.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:26:31.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:31.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:26:32.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:33.133 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:26:33.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:26:33.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:33.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:26:33.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:34.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:34.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:34.483 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:26:34.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:26:34.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:34.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:26:34.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:35.819 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:26:35.788+0000 7f419c84df40 -1 osd.3 54 log_to_monitors {default=true} 2021-07-26T06:26:35.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:35.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:35.846 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:26:35.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:26:35.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:26:35.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:36.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:26:37.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:26:37.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:26:37.404 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:26:37.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:26:37.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:26:37.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:26:38.170 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 67 up_thru 67 down_at 55 last_clean_interval [46,54) [v2:127.0.0.1:6826/118749,v1:127.0.0.1:6827/118749] [v2:127.0.0.1:6828/118749,v1:127.0.0.1:6829/118749] exists,up 93101700-5763-487e-b234-f4cc2d81e938 2021-07-26T06:26:38.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:26:38.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:26:38.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:26:38.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-07-26T06:26:38.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=118749 2021-07-26T06:26:38.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-07-26T06:26:38.233 INFO:tasks.workunit.client.0.smithi094.stdout:hints = 1 2021-07-26T06:26:38.233 INFO:tasks.workunit.client.0.smithi094.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-07-26T06:26:38.234 INFO:tasks.workunit.client.0.smithi094.stdout:Object prefix: benchmark_data_smithi094_119724 2021-07-26T06:26:38.234 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:26:38.234 INFO:tasks.workunit.client.0.smithi094.stdout: 0 0 0 0 0 0 - 0 2021-07-26T06:26:39.206 INFO:tasks.workunit.client.0.smithi094.stdout: 1 16 36 20 0.078117 0.078125 0.266527 0.278739 2021-07-26T06:26:40.209 INFO:tasks.workunit.client.0.smithi094.stdout: 2 16 38 22 0.0429634 0.0078125 0.0918002 0.263649 2021-07-26T06:26:41.206 INFO:tasks.workunit.client.0.smithi094.stdout: 3 16 38 22 0.0286419 0 - 0.263649 2021-07-26T06:26:42.206 INFO:tasks.workunit.client.0.smithi094.stdout: 4 16 38 22 0.0214814 0 - 0.263649 2021-07-26T06:26:43.206 INFO:tasks.workunit.client.0.smithi094.stdout: 5 16 38 22 0.017185 0 - 0.263649 2021-07-26T06:26:44.207 INFO:tasks.workunit.client.0.smithi094.stdout: 6 16 38 22 0.0143208 0 - 0.263649 2021-07-26T06:26:45.206 INFO:tasks.workunit.client.0.smithi094.stdout: 7 16 39 23 0.0128329 0.00078125 6.1061 0.517669 2021-07-26T06:26:46.206 INFO:tasks.workunit.client.0.smithi094.stdout: 8 16 44 28 0.0136698 0.0195312 0.110339 0.453782 2021-07-26T06:26:47.206 INFO:tasks.workunit.client.0.smithi094.stdout: 9 16 65 49 0.0212642 0.0820312 0.257345 2.85206 2021-07-26T06:26:48.209 INFO:tasks.workunit.client.0.smithi094.stdout: 10 16 120 104 0.040619 0.214844 0.240313 1.52208 2021-07-26T06:26:49.207 INFO:tasks.workunit.client.0.smithi094.stdout: 11 16 169 153 0.0543243 0.191406 0.3313 1.13829 2021-07-26T06:26:50.207 INFO:tasks.workunit.client.0.smithi094.stdout: 12 16 206 190 0.0618399 0.144531 0.317994 0.996751 2021-07-26T06:26:51.207 INFO:tasks.workunit.client.0.smithi094.stdout: 13 16 252 236 0.0709031 0.179688 0.342666 0.871009 2021-07-26T06:26:52.210 INFO:tasks.workunit.client.0.smithi094.stdout: 14 16 303 287 0.0800666 0.199219 0.327921 0.77173 2021-07-26T06:26:53.210 INFO:tasks.workunit.client.0.smithi094.stdout: 15 16 353 337 0.087748 0.195312 0.347248 0.708058 2021-07-26T06:26:54.208 INFO:tasks.workunit.client.0.smithi094.stdout: 16 16 396 380 0.0927604 0.167969 0.349149 0.664543 2021-07-26T06:26:55.208 INFO:tasks.workunit.client.0.smithi094.stdout: 17 16 446 430 0.098791 0.195312 0.325365 0.625968 2021-07-26T06:26:56.209 INFO:tasks.workunit.client.0.smithi094.stdout: 18 16 492 476 0.103284 0.179688 0.268448 0.597063 2021-07-26T06:26:57.213 INFO:tasks.workunit.client.0.smithi094.stdout: 19 16 529 513 0.105453 0.144531 0.166737 0.584142 2021-07-26T06:26:58.209 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:26:58.185275+0000 min lat: 0.0841101 max lat: 8.66992 avg lat: 0.557568 2021-07-26T06:26:58.210 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:26:58.210 INFO:tasks.workunit.client.0.smithi094.stdout: 20 16 584 568 0.110921 0.214844 0.352775 0.557568 2021-07-26T06:26:59.209 INFO:tasks.workunit.client.0.smithi094.stdout: 21 16 637 621 0.115496 0.207031 0.235036 0.536592 2021-07-26T06:27:00.212 INFO:tasks.workunit.client.0.smithi094.stdout: 22 16 689 673 0.119478 0.203125 0.194765 0.520181 2021-07-26T06:27:01.209 INFO:tasks.workunit.client.0.smithi094.stdout: 23 16 740 724 0.122944 0.199219 0.303801 0.50427 2021-07-26T06:27:02.209 INFO:tasks.workunit.client.0.smithi094.stdout: 24 16 788 772 0.125632 0.1875 0.321752 0.495398 2021-07-26T06:27:03.210 INFO:tasks.workunit.client.0.smithi094.stdout: 25 16 833 817 0.127637 0.175781 0.346371 0.485703 2021-07-26T06:27:04.209 INFO:tasks.workunit.client.0.smithi094.stdout: 26 16 878 862 0.129488 0.175781 0.390496 0.479337 2021-07-26T06:27:05.210 INFO:tasks.workunit.client.0.smithi094.stdout: 27 16 923 907 0.131201 0.175781 0.185616 0.471672 2021-07-26T06:27:06.210 INFO:tasks.workunit.client.0.smithi094.stdout: 28 16 970 954 0.133072 0.183594 0.391177 0.466466 2021-07-26T06:27:07.211 INFO:tasks.workunit.client.0.smithi094.stdout: 29 16 1016 1000 0.134678 0.179688 0.368536 0.461394 2021-07-26T06:27:08.210 INFO:tasks.workunit.client.0.smithi094.stdout: 30 16 1063 1047 0.136308 0.183594 0.336711 0.456462 2021-07-26T06:27:09.210 INFO:tasks.workunit.client.0.smithi094.stdout:Total time run: 30.2139 2021-07-26T06:27:09.211 INFO:tasks.workunit.client.0.smithi094.stdout:Total writes made: 1063 2021-07-26T06:27:09.211 INFO:tasks.workunit.client.0.smithi094.stdout:Write size: 4096 2021-07-26T06:27:09.211 INFO:tasks.workunit.client.0.smithi094.stdout:Object size: 4096 2021-07-26T06:27:09.212 INFO:tasks.workunit.client.0.smithi094.stdout:Bandwidth (MB/sec): 0.137431 2021-07-26T06:27:09.212 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Bandwidth: 0.0803939 2021-07-26T06:27:09.212 INFO:tasks.workunit.client.0.smithi094.stdout:Max bandwidth (MB/sec): 0.214844 2021-07-26T06:27:09.212 INFO:tasks.workunit.client.0.smithi094.stdout:Min bandwidth (MB/sec): 0 2021-07-26T06:27:09.213 INFO:tasks.workunit.client.0.smithi094.stdout:Average IOPS: 35 2021-07-26T06:27:09.213 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev IOPS: 20.5925 2021-07-26T06:27:09.213 INFO:tasks.workunit.client.0.smithi094.stdout:Max IOPS: 55 2021-07-26T06:27:09.213 INFO:tasks.workunit.client.0.smithi094.stdout:Min IOPS: 0 2021-07-26T06:27:09.213 INFO:tasks.workunit.client.0.smithi094.stdout:Average Latency(s): 0.453509 2021-07-26T06:27:09.214 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Latency(s): 0.960031 2021-07-26T06:27:09.214 INFO:tasks.workunit.client.0.smithi094.stdout:Max latency(s): 8.66992 2021-07-26T06:27:09.214 INFO:tasks.workunit.client.0.smithi094.stdout:Min latency(s): 0.0841101 2021-07-26T06:27:09.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 115675 2021-07-26T06:27:09.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2021-07-26T06:27:10.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 115675 2021-07-26T06:27:10.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (115675) - No such process 2021-07-26T06:27:10.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2021-07-26T06:27:10.875 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:27:10.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 116694 2021-07-26T06:27:10.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2021-07-26T06:27:11.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 116694 2021-07-26T06:27:11.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (116694) - No such process 2021-07-26T06:27:11.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2021-07-26T06:27:12.389 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T06:27:12.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 117730 2021-07-26T06:27:12.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2021-07-26T06:27:13.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 117730 2021-07-26T06:27:13.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (117730) - No such process 2021-07-26T06:27:13.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2021-07-26T06:27:13.833 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already down. 2021-07-26T06:27:13.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 118749 2021-07-26T06:27:13.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2021-07-26T06:27:14.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 118749 2021-07-26T06:27:14.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (118749) - No such process 2021-07-26T06:27:14.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2021-07-26T06:27:15.360 INFO:tasks.workunit.client.0.smithi094.stderr:osd.3 is already down. 2021-07-26T06:27:15.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:27:17.172 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:17.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2021-07-26T06:27:17.710 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records in 2021-07-26T06:27:17.710 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records out 2021-07-26T06:27:17.711 INFO:tasks.workunit.client.0.smithi094.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.531291 s, 2.0 GB/s 2021-07-26T06:27:17.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2021-07-26T06:27:17.728 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:27:27.267 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2021-07-26T06:27:27.267 INFO:tasks.workunit.client.0.smithi094.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2021-07-26T06:27:27.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:27:29.083 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:29.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2021-07-26T06:27:29.375 INFO:tasks.workunit.client.0.smithi094.stderr:512+0 records in 2021-07-26T06:27:29.375 INFO:tasks.workunit.client.0.smithi094.stderr:512+0 records out 2021-07-26T06:27:29.375 INFO:tasks.workunit.client.0.smithi094.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.284706 s, 1.9 GB/s 2021-07-26T06:27:29.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2021-07-26T06:27:29.394 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:27:35.184 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-07-26T06:27:35.185 INFO:tasks.workunit.client.0.smithi094.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2021-07-26T06:27:35.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2021-07-26T06:27:35.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-07-26T06:27:37.050 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:37.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:27:38.870 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:38.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2021-07-26T06:27:39.403 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records in 2021-07-26T06:27:39.404 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records out 2021-07-26T06:27:39.404 INFO:tasks.workunit.client.0.smithi094.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.52585 s, 2.0 GB/s 2021-07-26T06:27:39.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2021-07-26T06:27:39.420 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:27:49.210 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-07-26T06:27:49.210 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2021-07-26T06:27:49.210 INFO:tasks.workunit.client.0.smithi094.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2021-07-26T06:27:49.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2021-07-26T06:27:49.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:27:50.637 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:50.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-07-26T06:27:50.668 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:27:51.949 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-07-26T06:27:51.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2021-07-26T06:27:52.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-07-26T06:27:52.861 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:52.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2021-07-26T06:27:54.681 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:54.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-07-26T06:27:55.217 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records in 2021-07-26T06:27:55.218 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records out 2021-07-26T06:27:55.218 INFO:tasks.workunit.client.0.smithi094.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.528421 s, 2.0 GB/s 2021-07-26T06:27:55.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-07-26T06:27:55.237 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:27:55.242 INFO:tasks.workunit.client.0.smithi094.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2021-07-26T06:27:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2021-07-26T06:27:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-07-26T06:27:57.057 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:27:57.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-07-26T06:28:05.697 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records in 2021-07-26T06:28:05.697 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records out 2021-07-26T06:28:05.698 INFO:tasks.workunit.client.0.smithi094.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.51677 s, 126 MB/s 2021-07-26T06:28:05.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-07-26T06:28:05.716 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:28:07.764 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-07-26T06:28:07.765 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-07-26T06:28:07.765 INFO:tasks.workunit.client.0.smithi094.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2021-07-26T06:28:07.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-07-26T06:28:09.076 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:28:09.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2021-07-26T06:28:09.611 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records in 2021-07-26T06:28:09.611 INFO:tasks.workunit.client.0.smithi094.stderr:1024+0 records out 2021-07-26T06:28:09.612 INFO:tasks.workunit.client.0.smithi094.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.527319 s, 2.0 GB/s 2021-07-26T06:28:09.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2021-07-26T06:28:09.629 INFO:tasks.workunit.client.0.smithi094.stdout:inferring bluefs devices from bluestore path 2021-07-26T06:28:18.684 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2021-07-26T06:28:18.685 INFO:tasks.workunit.client.0.smithi094.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-07-26T06:28:18.685 INFO:tasks.workunit.client.0.smithi094.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2021-07-26T06:28:18.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-07-26T06:28:20.004 INFO:tasks.workunit.client.0.smithi094.stdout:fsck success 2021-07-26T06:28:20.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-07-26T06:28:20.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:28:20.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:28:20.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:28:20.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:28:20.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-07-26T06:28:20.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:28:20.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:28:20.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:28:20.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:28:20.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-07-26T06:28:20.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-07-26T06:28:20.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:28:20.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:28:20.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:28:20.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:28:20.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:28:20.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:20.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:20.017 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:20.018 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:20.018 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:28:20.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:20.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:20.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:28:20.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:28:20.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:28:20.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:28:20.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:28:20.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:20.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:28:20.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:28:20.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-07-26T06:28:20.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:28:20.021 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:28:20.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:20.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-07-26T06:28:20.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:28:20.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:28:20.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:28:20.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:28:20.046 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:20.021+0000 7f7c474faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:20.049 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:20.025+0000 7f7c474faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:20.052 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:20.029+0000 7f7c474faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:20.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:28:20.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:28:20.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:28:20.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:28:20.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:28:20.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:20.372 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:28:20.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:28:20.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:20.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:20.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:21.086 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:21.061+0000 7f7c474faf40 -1 Falling back to public interface 2021-07-26T06:28:21.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:21.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:21.730 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:28:21.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:28:21.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:21.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:22.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:23.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:23.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:23.123 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:28:23.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:28:23.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:23.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:23.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:24.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:24.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:24.480 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:28:24.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:28:24.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:24.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:24.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:25.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:25.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:25.850 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:28:25.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:28:25.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:25.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:26.014 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:25.989+0000 7f7c474faf40 -1 osd.0 69 log_to_monitors {default=true} 2021-07-26T06:28:26.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:27.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:27.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:27.253 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T06:28:27.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 5 2021-07-26T06:28:27.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:27.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:28:27.613 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 80 up_thru 80 down_at 70 last_clean_interval [57,69) [v2:127.0.0.1:6802/125993,v1:127.0.0.1:6803/125993] [v2:127.0.0.1:6804/125993,v1:127.0.0.1:6805/125993] exists,up 48983ae5-ea84-4702-9a7d-c6e04ee74555 2021-07-26T06:28:27.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:28:27.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:28:27.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:28:27.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-07-26T06:28:27.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=125993 2021-07-26T06:28:27.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-07-26T06:28:27.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:28:27.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:28:27.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:28:27.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:28:27.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-07-26T06:28:27.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:28:27.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:28:27.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:28:27.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:28:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-07-26T06:28:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-07-26T06:28:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:28:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:28:27.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:28:27.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:28:27.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:28:27.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:27.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:27.629 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:27.629 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:27.629 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:28:27.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:27.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:27.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:28:27.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:28:27.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:28:27.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:28:27.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:28:27.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:27.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:28:27.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:28:27.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-07-26T06:28:27.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:28:27.633 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:28:27.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:27.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-07-26T06:28:27.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:28:27.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:28:27.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:28:27.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:28:27.644 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:27.621+0000 7f0a3fe2cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:27.701 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:27.677+0000 7f0a3fe2cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:27.703 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:27.681+0000 7f0a3fe2cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:28.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:28:28.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:28:28.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:28:28.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:28:28.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:28:28.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:28.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:28:28.011 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:28:28.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:28.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:28.245 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:28.221+0000 7f0a3fe2cf40 -1 Falling back to public interface 2021-07-26T06:28:28.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:29.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:29.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:29.359 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:28:29.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:28:29.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:29.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:29.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:30.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:30.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:30.762 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:28:30.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:28:30.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:30.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:31.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:32.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:32.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:32.111 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:28:32.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:28:32.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:32.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:32.320 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:32.297+0000 7f0a3fe2cf40 -1 osd.1 72 log_to_monitors {default=true} 2021-07-26T06:28:32.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:33.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:33.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:33.659 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:28:33.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:28:33.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:33.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:28:34.158 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 73 last_clean_interval [60,72) [v2:127.0.0.1:6810/127013,v1:127.0.0.1:6811/127013] [v2:127.0.0.1:6812/127013,v1:127.0.0.1:6813/127013] exists,up 817d3194-fc16-4662-9b83-ea4d7aabf413 2021-07-26T06:28:34.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:28:34.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:28:34.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:28:34.160 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-07-26T06:28:34.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=127013 2021-07-26T06:28:34.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-07-26T06:28:34.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:28:34.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:28:34.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:28:34.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:28:34.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-07-26T06:28:34.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:28:34.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:28:34.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:28:34.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:28:34.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-07-26T06:28:34.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-07-26T06:28:34.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:28:34.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:28:34.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:28:34.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:28:34.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:28:34.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:34.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:34.171 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:34.171 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:34.172 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:28:34.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:34.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:34.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:28:34.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:28:34.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:28:34.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:28:34.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:28:34.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:34.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:28:34.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:28:34.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-07-26T06:28:34.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:28:34.175 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:28:34.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:34.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-07-26T06:28:34.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:28:34.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:28:34.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:28:34.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:28:34.192 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:34.169+0000 7f4967deff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:34.195 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:34.169+0000 7f4967deff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:34.197 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:34.173+0000 7f4967deff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:34.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:28:34.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:28:34.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:28:34.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:28:34.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:28:34.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:34.552 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:28:34.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:28:34.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:34.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:28:34.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:35.001 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:34.977+0000 7f4967deff40 -1 Falling back to public interface 2021-07-26T06:28:35.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:35.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:35.976 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:28:35.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:28:35.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:35.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:28:36.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:37.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:37.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:37.338 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:28:37.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:28:37.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:37.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:28:37.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:38.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:38.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:38.784 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:28:38.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:28:38.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:38.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:28:38.928 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:38.905+0000 7f4967deff40 -1 osd.2 75 log_to_monitors {default=true} 2021-07-26T06:28:39.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:40.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:40.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:40.294 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:28:40.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:28:40.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:40.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:28:40.845 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 87 up_thru 87 down_at 76 last_clean_interval [63,75) [v2:127.0.0.1:6818/127983,v1:127.0.0.1:6819/127983] [v2:127.0.0.1:6820/127983,v1:127.0.0.1:6821/127983] exists,up 47d572b5-8285-4254-ab35-52b7fa3c0ddf 2021-07-26T06:28:40.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:28:40.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:28:40.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:28:40.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-07-26T06:28:40.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=127983 2021-07-26T06:28:40.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-07-26T06:28:40.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:28:40.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:28:40.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=3 2021-07-26T06:28:40.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:28:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-07-26T06:28:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-07-26T06:28:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:28:40.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:28:40.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:28:40.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-07-26T06:28:40.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-07-26T06:28:40.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:28:40.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:28:40.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:28:40.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-07-26T06:28:40.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:28:40.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:28:40.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:28:40.856 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:28:40.857 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:28:40.857 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:28:40.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:40.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' 2021-07-26T06:28:40.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:28:40.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-07-26T06:28:40.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-07-26T06:28:40.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:28:40.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:28:40.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:40.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:28:40.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:28:40.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-07-26T06:28:40.860 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T06:28:40.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.3 2021-07-26T06:28:40.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 3 --fsid=f2c2a957-1b45-47b7-a9c0-2b66c3007a3c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.83476/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:28:40.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-07-26T06:28:40.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 3 = 3 ']' 2021-07-26T06:28:40.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:28:40.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:28:40.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:28:40.879 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:40.857+0000 7f245eeb0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:40.886 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:40.861+0000 7f245eeb0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:40.888 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:40.865+0000 7f245eeb0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:28:41.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 3 2021-07-26T06:28:41.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:28:41.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:28:41.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:28:41.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:28:41.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:41.246 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:28:41.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:28:41.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:41.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:28:41.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:41.677 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:41.653+0000 7f245eeb0f40 -1 Falling back to public interface 2021-07-26T06:28:42.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:42.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:42.661 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:28:42.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:28:42.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:42.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:28:43.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:44.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:44.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:44.042 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:28:44.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:28:44.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:44.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:28:44.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:45.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:45.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:45.409 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:28:45.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:28:45.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:45.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:28:45.574 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:28:45.549+0000 7f245eeb0f40 -1 osd.3 77 log_to_monitors {default=true} 2021-07-26T06:28:45.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:28:46.884 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:28:46.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:28:46.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:28:46.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 4 2021-07-26T06:28:46.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:28:46.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:28:47.530 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 91 up_thru 91 down_at 78 last_clean_interval [67,77) [v2:127.0.0.1:6826/128965,v1:127.0.0.1:6827/128965] [v2:127.0.0.1:6828/128965,v1:127.0.0.1:6829/128965] exists,up 93101700-5763-487e-b234-f4cc2d81e938 2021-07-26T06:28:47.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:28:47.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:28:47.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:28:47.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-07-26T06:28:47.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=128965 2021-07-26T06:28:47.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-07-26T06:28:47.646 INFO:tasks.workunit.client.0.smithi094.stdout:hints = 1 2021-07-26T06:28:47.646 INFO:tasks.workunit.client.0.smithi094.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-07-26T06:28:47.646 INFO:tasks.workunit.client.0.smithi094.stdout:Object prefix: benchmark_data_smithi094_129932 2021-07-26T06:28:47.647 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:28:47.647 INFO:tasks.workunit.client.0.smithi094.stdout: 0 0 0 0 0 0 - 0 2021-07-26T06:28:48.646 INFO:tasks.workunit.client.0.smithi094.stdout: 1 16 43 27 0.105455 0.105469 0.259017 0.20426 2021-07-26T06:28:49.646 INFO:tasks.workunit.client.0.smithi094.stdout: 2 16 58 42 0.0820195 0.0585938 0.112332 0.178659 2021-07-26T06:28:50.646 INFO:tasks.workunit.client.0.smithi094.stdout: 3 16 61 45 0.0585852 0.0117188 0.0673259 0.177443 2021-07-26T06:28:51.646 INFO:tasks.workunit.client.0.smithi094.stdout: 4 16 61 45 0.0439386 0 - 0.177443 2021-07-26T06:28:52.646 INFO:tasks.workunit.client.0.smithi094.stdout: 5 16 61 45 0.035151 0 - 0.177443 2021-07-26T06:28:53.646 INFO:tasks.workunit.client.0.smithi094.stdout: 6 16 61 45 0.0292925 0 - 0.177443 2021-07-26T06:28:54.647 INFO:tasks.workunit.client.0.smithi094.stdout: 7 16 61 45 0.0251079 0 - 0.177443 2021-07-26T06:28:55.647 INFO:tasks.workunit.client.0.smithi094.stdout: 8 16 61 45 0.0219693 0 - 0.177443 2021-07-26T06:28:56.647 INFO:tasks.workunit.client.0.smithi094.stdout: 9 16 78 62 0.0269057 0.0110677 0.154023 1.87935 2021-07-26T06:28:57.647 INFO:tasks.workunit.client.0.smithi094.stdout: 10 16 117 101 0.0394473 0.152344 0.153456 1.24501 2021-07-26T06:28:58.647 INFO:tasks.workunit.client.0.smithi094.stdout: 11 16 195 179 0.0635562 0.304688 0.230667 0.970272 2021-07-26T06:28:59.647 INFO:tasks.workunit.client.0.smithi094.stdout: 12 16 273 257 0.0836465 0.304688 0.203264 0.740269 2021-07-26T06:29:00.647 INFO:tasks.workunit.client.0.smithi094.stdout: 13 16 334 318 0.0955387 0.238281 0.22708 0.64545 2021-07-26T06:29:01.647 INFO:tasks.workunit.client.0.smithi094.stdout: 14 16 408 392 0.109359 0.289062 0.0994437 0.567271 2021-07-26T06:29:02.648 INFO:tasks.workunit.client.0.smithi094.stdout: 15 16 475 459 0.119513 0.261719 0.225691 0.519922 2021-07-26T06:29:03.648 INFO:tasks.workunit.client.0.smithi094.stdout: 16 16 545 529 0.129131 0.273438 0.148334 0.479317 2021-07-26T06:29:04.648 INFO:tasks.workunit.client.0.smithi094.stdout: 17 16 625 609 0.139915 0.3125 0.0861982 0.442735 2021-07-26T06:29:05.648 INFO:tasks.workunit.client.0.smithi094.stdout: 18 16 699 683 0.148198 0.289062 0.213655 0.419643 2021-07-26T06:29:06.648 INFO:tasks.workunit.client.0.smithi094.stdout: 19 16 777 761 0.156432 0.304688 0.217327 0.397676 2021-07-26T06:29:07.648 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:29:07.630594+0000 min lat: 0.0647701 max lat: 9.63622 avg lat: 0.381746 2021-07-26T06:29:07.648 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:29:07.649 INFO:tasks.workunit.client.0.smithi094.stdout: 20 16 850 834 0.162866 0.285156 0.125534 0.381746 2021-07-26T06:29:08.648 INFO:tasks.workunit.client.0.smithi094.stdout: 21 16 913 897 0.166828 0.246094 0.181951 0.371475 2021-07-26T06:29:09.648 INFO:tasks.workunit.client.0.smithi094.stdout: 22 16 989 973 0.172737 0.296875 0.206359 0.360227 2021-07-26T06:29:10.648 INFO:tasks.workunit.client.0.smithi094.stdout: 23 16 1067 1051 0.178472 0.304688 0.110261 0.348282 2021-07-26T06:29:11.648 INFO:tasks.workunit.client.0.smithi094.stdout: 24 16 1133 1117 0.181776 0.257812 0.141074 0.34214 2021-07-26T06:29:12.649 INFO:tasks.workunit.client.0.smithi094.stdout: 25 16 1213 1197 0.187001 0.3125 0.247301 0.333188 2021-07-26T06:29:13.649 INFO:tasks.workunit.client.0.smithi094.stdout: 26 16 1276 1260 0.189272 0.246094 0.196499 0.328634 2021-07-26T06:29:14.650 INFO:tasks.workunit.client.0.smithi094.stdout: 27 16 1349 1333 0.192822 0.285156 0.139277 0.322876 2021-07-26T06:29:15.650 INFO:tasks.workunit.client.0.smithi094.stdout: 28 16 1401 1385 0.193189 0.203125 0.409424 0.321222 2021-07-26T06:29:16.649 INFO:tasks.workunit.client.0.smithi094.stdout: 29 16 1473 1457 0.196224 0.28125 0.26572 0.317181 2021-07-26T06:29:17.650 INFO:tasks.workunit.client.0.smithi094.stdout: 30 16 1548 1532 0.199448 0.292969 0.212957 0.312695 2021-07-26T06:29:18.650 INFO:tasks.workunit.client.0.smithi094.stdout:Total time run: 30.2463 2021-07-26T06:29:18.650 INFO:tasks.workunit.client.0.smithi094.stdout:Total writes made: 1548 2021-07-26T06:29:18.650 INFO:tasks.workunit.client.0.smithi094.stdout:Write size: 4096 2021-07-26T06:29:18.651 INFO:tasks.workunit.client.0.smithi094.stdout:Object size: 4096 2021-07-26T06:29:18.651 INFO:tasks.workunit.client.0.smithi094.stdout:Bandwidth (MB/sec): 0.199921 2021-07-26T06:29:18.651 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Bandwidth: 0.123818 2021-07-26T06:29:18.651 INFO:tasks.workunit.client.0.smithi094.stdout:Max bandwidth (MB/sec): 0.3125 2021-07-26T06:29:18.652 INFO:tasks.workunit.client.0.smithi094.stdout:Min bandwidth (MB/sec): 0 2021-07-26T06:29:18.652 INFO:tasks.workunit.client.0.smithi094.stdout:Average IOPS: 51 2021-07-26T06:29:18.652 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev IOPS: 31.7411 2021-07-26T06:29:18.652 INFO:tasks.workunit.client.0.smithi094.stdout:Max IOPS: 80 2021-07-26T06:29:18.652 INFO:tasks.workunit.client.0.smithi094.stdout:Min IOPS: 0 2021-07-26T06:29:18.653 INFO:tasks.workunit.client.0.smithi094.stdout:Average Latency(s): 0.312123 2021-07-26T06:29:18.653 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Latency(s): 0.848283 2021-07-26T06:29:18.653 INFO:tasks.workunit.client.0.smithi094.stdout:Max latency(s): 9.63622 2021-07-26T06:29:18.653 INFO:tasks.workunit.client.0.smithi094.stdout:Min latency(s): 0.0647701 2021-07-26T06:29:18.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2021-07-26T06:29:18.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:29:18.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:29:18.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:29:18.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:29:18.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:29:18.725 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:29:18.725 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:29:18.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:29:18.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:29:18.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:29:18.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:29:18.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:29:18.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:29:18.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:29:18.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:29:19.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:29:19.293 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:29:19.293 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:29:19.294 INFO:tasks.workunit.client.0.smithi094.stderr:3' 2021-07-26T06:29:19.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:29:19.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:19.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:29:19.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=343597383693 2021-07-26T06:29:19.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 343597383693 2021-07-26T06:29:19.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-343597383693' 2021-07-26T06:29:19.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:19.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:29:19.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=356482285580 2021-07-26T06:29:19.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 356482285580 2021-07-26T06:29:19.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-343597383693 1-356482285580' 2021-07-26T06:29:19.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:19.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:29:19.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=373662154763 2021-07-26T06:29:19.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 373662154763 2021-07-26T06:29:19.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-343597383693 1-356482285580 2-373662154763' 2021-07-26T06:29:19.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:29:19.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:29:19.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=390842023945 2021-07-26T06:29:19.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 390842023945 2021-07-26T06:29:19.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-343597383693 1-356482285580 2-373662154763 3-390842023945' 2021-07-26T06:29:19.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:19.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-343597383693 2021-07-26T06:29:19.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:19.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:29:19.866 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-343597383693 2021-07-26T06:29:19.867 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:19.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=343597383693 2021-07-26T06:29:19.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 343597383693' 2021-07-26T06:29:19.869 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 343597383693 2021-07-26T06:29:19.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:29:20.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 343597383693 -lt 343597383693 2021-07-26T06:29:20.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:20.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-356482285580 2021-07-26T06:29:20.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:20.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:29:20.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-356482285580 2021-07-26T06:29:20.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:20.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=356482285580 2021-07-26T06:29:20.237 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 356482285580 2021-07-26T06:29:20.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 356482285580' 2021-07-26T06:29:20.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:29:20.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 356482285579 -lt 356482285580 2021-07-26T06:29:20.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:29:21.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:29:21.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:29:21.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 356482285580 -lt 356482285580 2021-07-26T06:29:21.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:21.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-373662154763 2021-07-26T06:29:21.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:21.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:29:21.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-373662154763 2021-07-26T06:29:21.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:21.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=373662154763 2021-07-26T06:29:21.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 373662154763' 2021-07-26T06:29:21.996 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 373662154763 2021-07-26T06:29:21.997 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:29:22.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 373662154763 -lt 373662154763 2021-07-26T06:29:22.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:29:22.353 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-390842023945 2021-07-26T06:29:22.354 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:29:22.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:29:22.356 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:29:22.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-390842023945 2021-07-26T06:29:22.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=390842023945 2021-07-26T06:29:22.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 390842023945' 2021-07-26T06:29:22.358 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 390842023945 2021-07-26T06:29:22.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:29:22.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 390842023946 -lt 390842023945 2021-07-26T06:29:22.755 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:29:22.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:29:22.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:29:23.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 16 == 0 2021-07-26T06:29:23.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:29:23.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:29:23.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:29:23.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:29:23.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:29:23.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:29:23.263 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:29:23.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=16 2021-07-26T06:29:23.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:29:23.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:29:23.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:29:24.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 16 = 16 2021-07-26T06:29:24.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:29:24.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:29:24.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:25: run: teardown td/osd-bluefs-volume-ops 2021-07-26T06:29:24.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:29:24.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:24.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-07-26T06:29:24.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:24.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:24.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:24.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:24.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:24.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:24.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:24.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:24.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:24.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:24.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:24.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:24.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:24.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:24.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:24.311 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:24.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:24.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:24.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-07-26T06:29:24.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:24.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:24.697 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:29:24.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83476 2021-07-26T06:29:24.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:24.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:24.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:29:24.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-bluefs-volume-ops 0 2021-07-26T06:29:24.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-07-26T06:29:24.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:29:24.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-07-26T06:29:24.700 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:24.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:24.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:24.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:24.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:24.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:24.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:24.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:24.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:24.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:24.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:24.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:24.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:24.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:24.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:24.711 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:24.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:24.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:29:24.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-07-26T06:29:24.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:24.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:24.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83476 2021-07-26T06:29:24.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83476 2021-07-26T06:29:24.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:24.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:24.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:29:24.719 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:29:24.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:29:24.733 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2021-07-26T06:29:24.734 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2021-07-26T06:29:24.794 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:29:24.805 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:29:24.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:29:24.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:29:24.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:29:24.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:29:24.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:29:24.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:29:24.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:29:24.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:29:24.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-config 2021-07-26T06:29:24.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2021-07-26T06:29:24.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2021-07-26T06:29:24.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2021-07-26T06:29:24.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2021-07-26T06:29:24.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2021-07-26T06:29:24.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2021-07-26T06:29:24.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=375cb885-9b5c-4170-b633-98ddc7448e39 --auth-supported=none ' 2021-07-26T06:29:24.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2021-07-26T06:29:24.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2021-07-26T06:29:24.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:29:24.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2021-07-26T06:29:24.812 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_config_track 2021-07-26T06:29:24.812 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_default_adjustment' 2021-07-26T06:29:24.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-07-26T06:29:24.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-07-26T06:29:24.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-07-26T06:29:24.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-07-26T06:29:24.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-07-26T06:29:24.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:24.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-07-26T06:29:24.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:24.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:24.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:24.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:24.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:24.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:24.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:24.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:24.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:24.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:24.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:24.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:24.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:24.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:24.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:24.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:24.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:24.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:24.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-07-26T06:29:24.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:24.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:24.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:24.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.130461 2021-07-26T06:29:24.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:24.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:24.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-07-26T06:29:24.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:29:24.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:24.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:24.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.130461 2021-07-26T06:29:24.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:29:24.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:29:24.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:29:24.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:29:24.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-07-26T06:29:24.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2021-07-26T06:29:24.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2021-07-26T06:29:24.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2021-07-26T06:29:24.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-07-26T06:29:24.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:29:24.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:29:24.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:29:24.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-07-26T06:29:24.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-07-26T06:29:25.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:29:25.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:25.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:25.503 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:25.503 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:25.504 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:25.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:25.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:29:25.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:29:25.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:29:25.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:25.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:25.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:29:25.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:29:25.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:25.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:25.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:25.781 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:25.781 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:25.781 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:25.781 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.130461/ceph-mon.a.asok 2021-07-26T06:29:25.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:25.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.130461/ceph-mon.a.asok config get fsid 2021-07-26T06:29:25.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:29:25.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:25.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:25.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:29:25.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:29:25.871 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:25.871 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:25.872 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:25.872 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:25.872 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:25.872 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:25.873 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.130461/ceph-mon.a.asok 2021-07-26T06:29:25.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:25.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.130461/ceph-mon.a.asok config get mon_host 2021-07-26T06:29:25.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2021-07-26T06:29:25.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-07-26T06:29:25.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:29:25.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:29:25.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:29:25.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-07-26T06:29:25.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:29:26.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:29:26.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:26.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:26.838 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:26.839 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:26.839 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:26.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:26.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:26.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:26.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2021-07-26T06:29:26.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2021-07-26T06:29:26.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-07-26T06:29:26.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-07-26T06:29:26.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:29:26.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:29:26.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:29:26.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-07-26T06:29:26.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=375cb885-9b5c-4170-b633-98ddc7448e39 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-07-26T06:29:26.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:29:26.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:29:26.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:29:26.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-07-26T06:29:26.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-07-26T06:29:26.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:29:26.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:29:26.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:29:26.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-07-26T06:29:26.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:29:26.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:26.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:26.877 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:26.877 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:26.877 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:26.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:26.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:26.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:29:26.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:29:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:29:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-07-26T06:29:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-07-26T06:29:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:29:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:29:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:29:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2021-07-26T06:29:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2021-07-26T06:29:26.882 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:29:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ada4fbc8-2fcd-4cec-ad6c-03bd8c340c21 2021-07-26T06:29:26.882 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 ada4fbc8-2fcd-4cec-ad6c-03bd8c340c21 2021-07-26T06:29:26.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ada4fbc8-2fcd-4cec-ad6c-03bd8c340c21' 2021-07-26T06:29:26.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:29:26.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGVv5ghU8ONBAAVxnxAojqF+kt4IqmyJbU2w== 2021-07-26T06:29:26.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGVv5ghU8ONBAAVxnxAojqF+kt4IqmyJbU2w=="}' 2021-07-26T06:29:26.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ada4fbc8-2fcd-4cec-ad6c-03bd8c340c21 -i td/osd-config/0/new.json 2021-07-26T06:29:27.358 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:29:27.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2021-07-26T06:29:27.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=375cb885-9b5c-4170-b633-98ddc7448e39 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQBGVv5ghU8ONBAAVxnxAojqF+kt4IqmyJbU2w== --osd-uuid ada4fbc8-2fcd-4cec-ad6c-03bd8c340c21 2021-07-26T06:29:27.397 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:27.378+0000 7fbdeb64bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:27.412 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:27.390+0000 7fbdeb64bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:27.414 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:27.394+0000 7fbdeb64bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:27.415 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:27.394+0000 7fbdeb64bf40 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-07-26T06:29:27.415 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:27.394+0000 7fbdeb64bf40 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-07-26T06:29:27.415 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:27.394+0000 7fbdeb64bf40 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-07-26T06:29:27.459 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:27.438+0000 7fbdeb64bf40 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-07-26T06:29:33.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2021-07-26T06:29:33.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:29:33.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:29:33.349 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:29:33.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:29:33.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:29:33.874 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:29:33.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=375cb885-9b5c-4170-b633-98ddc7448e39 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-07-26T06:29:33.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:29:33.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:29:33.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:29:33.904 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:33.882+0000 7f7b79837f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:33.910 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:33.890+0000 7f7b79837f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:33.914 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:33.890+0000 7f7b79837f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:34.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:29:34.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:29:34.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:29:34.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:29:34.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:29:34.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:34.241 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:29:34.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:29:34.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:34.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:34.447 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:34.426+0000 7f7b79837f40 -1 Falling back to public interface 2021-07-26T06:29:34.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:35.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:35.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:35.611 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:29:35.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:29:35.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:35.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:36.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:37.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:37.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:37.017 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:29:37.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:29:37.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:37.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:37.028 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:37.006+0000 7f7b79837f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:29:37.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:38.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:38.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:38.433 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:29:38.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:29:38.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:38.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:38.824 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/131565,v1:127.0.0.1:6803/131565] [v2:127.0.0.1:6804/131565,v1:127.0.0.1:6805/131565] exists,up ada4fbc8-2fcd-4cec-ad6c-03bd8c340c21 2021-07-26T06:29:38.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:29:38.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:29:38.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:29:38.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2021-07-26T06:29:38.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:29:38.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:29:38.827 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:38.827 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:38.827 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:38.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.130461/ceph-osd.0.asok 2021-07-26T06:29:38.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2021-07-26T06:29:38.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.130461/ceph-osd.0.asok log flush 2021-07-26T06:29:38.909 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:29:38.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-07-26T06:29:38.919 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:29:37.370+0000 7f7b79837f40 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-07-26T06:29:38.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-07-26T06:29:38.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-07-26T06:29:38.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:38.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-07-26T06:29:38.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:38.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:38.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:38.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:38.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:39.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:39.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:39.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:39.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:39.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:39.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:39.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:39.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:39.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:39.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:39.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:39.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:39.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:39.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-07-26T06:29:39.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:39.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:39.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:39.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.130461 2021-07-26T06:29:39.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:39.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:39.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-07-26T06:29:39.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-07-26T06:29:39.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-07-26T06:29:39.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-07-26T06:29:39.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-07-26T06:29:39.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:39.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-07-26T06:29:39.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:39.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:39.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:39.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:39.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:39.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:39.057 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:39.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:39.059 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:39.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:39.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:39.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:39.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:39.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:39.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:39.064 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:39.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:39.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:39.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-07-26T06:29:39.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:39.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:39.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:39.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.130461 2021-07-26T06:29:39.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:39.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:39.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-07-26T06:29:39.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:29:39.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:39.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:39.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.130461 2021-07-26T06:29:39.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:29:39.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:29:39.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:29:39.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-07-26T06:29:39.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2021-07-26T06:29:39.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2021-07-26T06:29:39.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2021-07-26T06:29:39.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-07-26T06:29:39.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:29:39.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:29:39.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:29:39.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-07-26T06:29:39.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-07-26T06:29:39.519 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:29:39.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:39.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:39.521 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:39.521 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:39.521 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:39.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:39.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:29:39.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:29:39.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:29:39.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:39.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:39.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:29:39.866 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:39.866 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:39.866 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:39.867 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:29:39.867 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:39.867 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:39.867 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:39.868 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.130461/ceph-mon.a.asok 2021-07-26T06:29:39.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:39.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.130461/ceph-mon.a.asok config get fsid 2021-07-26T06:29:39.956 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:29:39.956 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:39.956 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:39.956 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:29:39.958 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:29:39.958 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:39.958 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:39.959 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:39.959 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:39.959 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:39.959 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:39.960 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.130461/ceph-mon.a.asok 2021-07-26T06:29:39.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:39.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.130461/ceph-mon.a.asok config get mon_host 2021-07-26T06:29:40.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2021-07-26T06:29:40.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2021-07-26T06:29:40.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:29:40.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:29:40.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:29:40.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2021-07-26T06:29:40.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:29:40.943 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:29:40.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:40.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:40.944 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:40.945 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:40.945 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:40.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:40.946 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:40.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:40.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2021-07-26T06:29:40.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2021-07-26T06:29:40.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:29:40.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:29:40.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:29:40.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2021-07-26T06:29:40.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=375cb885-9b5c-4170-b633-98ddc7448e39 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-07-26T06:29:40.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:29:40.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:29:40.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:29:40.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-07-26T06:29:40.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-07-26T06:29:40.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:29:40.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:29:40.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:29:40.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-07-26T06:29:40.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:29:40.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:40.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:40.984 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:40.985 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:40.985 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:40.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:40.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' 2021-07-26T06:29:40.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:29:40.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:29:40.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:29:40.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-07-26T06:29:40.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-07-26T06:29:40.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:29:40.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:29:40.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:40.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:29:40.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:29:40.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2021-07-26T06:29:40.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:29:40.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=343f63cc-19fc-4fc2-a2cc-3198edbe9509 2021-07-26T06:29:40.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 343f63cc-19fc-4fc2-a2cc-3198edbe9509' 2021-07-26T06:29:40.991 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 343f63cc-19fc-4fc2-a2cc-3198edbe9509 2021-07-26T06:29:40.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:29:41.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUVv5gpx2qOhAATuZB+7f2q2ocZ70mgSWWYQ== 2021-07-26T06:29:41.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUVv5gpx2qOhAATuZB+7f2q2ocZ70mgSWWYQ=="}' 2021-07-26T06:29:41.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 343f63cc-19fc-4fc2-a2cc-3198edbe9509 -i td/osd-config/0/new.json 2021-07-26T06:29:41.352 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:29:41.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2021-07-26T06:29:41.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=375cb885-9b5c-4170-b633-98ddc7448e39 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBUVv5gpx2qOhAATuZB+7f2q2ocZ70mgSWWYQ== --osd-uuid 343f63cc-19fc-4fc2-a2cc-3198edbe9509 2021-07-26T06:29:41.394 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:41.374+0000 7fc7766d9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:41.396 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:41.378+0000 7fc7766d9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:41.397 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:41.378+0000 7fc7766d9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:41.378+0000 7fc7766d9f40 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-07-26T06:29:41.399 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:41.378+0000 7fc7766d9f40 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-07-26T06:29:41.399 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:41.378+0000 7fc7766d9f40 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-07-26T06:29:41.512 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:41.490+0000 7fc7766d9f40 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-07-26T06:29:47.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2021-07-26T06:29:47.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:29:47.405 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:29:47.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:29:47.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:29:47.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:29:47.870 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:29:47.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:29:47.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=375cb885-9b5c-4170-b633-98ddc7448e39 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.130461/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:47.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:29:47.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:29:47.898 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:47.878+0000 7face1b53f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:47.921 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:47.902+0000 7face1b53f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:47.924 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:47.902+0000 7face1b53f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:48.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:29:48.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:29:48.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:29:48.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:29:48.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:29:48.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:48.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:29:48.204 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:29:48.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:48.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:48.454 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:48.434+0000 7face1b53f40 -1 Falling back to public interface 2021-07-26T06:29:48.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:49.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:49.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:49.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:29:49.566 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:29:49.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:49.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:49.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:50.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:29:50.925 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:29:50.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:50.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:51.032 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:51.014+0000 7face1b53f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:29:51.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:29:52.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:29:52.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:29:52.350 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:29:52.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:29:52.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:29:52.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:29:52.706 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/133255,v1:127.0.0.1:6803/133255] [v2:127.0.0.1:6804/133255,v1:127.0.0.1:6805/133255] exists,up 343f63cc-19fc-4fc2-a2cc-3198edbe9509 2021-07-26T06:29:52.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:29:52.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:29:52.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:29:52.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2021-07-26T06:29:52.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2021-07-26T06:29:52.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2021-07-26T06:29:52.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2021-07-26T06:29:52.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2021-07-26T06:29:52.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2021-07-26T06:29:52.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-07-26T06:29:52.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2021-07-26T06:29:52.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2021-07-26T06:29:52.868 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:29:52.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2021-07-26T06:29:52.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:29:52.885 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:29:52.885 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:52.885 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:52.885 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:52.885 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.130461/ceph-osd.0.asok 2021-07-26T06:29:52.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2021-07-26T06:29:52.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.130461/ceph-osd.0.asok log flush 2021-07-26T06:29:52.969 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:29:52.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-07-26T06:29:52.979 INFO:tasks.workunit.client.0.smithi094.stdout:2021-07-26T06:29:52.834+0000 7face031d700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2021-07-26T06:29:52.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2021-07-26T06:29:52.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2021-07-26T06:29:52.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:29:52.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:29:52.983 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:52.983 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:52.983 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:52.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.130461/ceph-osd.0.asok 2021-07-26T06:29:52.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2021-07-26T06:29:52.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.130461/ceph-osd.0.asok log reopen 2021-07-26T06:29:53.069 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:29:53.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-07-26T06:29:53.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-07-26T06:29:53.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:53.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-07-26T06:29:53.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:53.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:53.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:53.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:53.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:53.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:53.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:53.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:53.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:53.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:53.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:53.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:53.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:53.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:53.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:53.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:53.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-07-26T06:29:53.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:53.210 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.210 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:53.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.130461 2021-07-26T06:29:53.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:53.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:53.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-07-26T06:29:53.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-07-26T06:29:53.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-07-26T06:29:53.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-07-26T06:29:53.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-07-26T06:29:53.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:53.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-07-26T06:29:53.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:53.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:53.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:53.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:53.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:53.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:53.218 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:53.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:53.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:53.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:53.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:53.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:53.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:53.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:53.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:53.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:53.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-07-26T06:29:53.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:53.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:53.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.130461 2021-07-26T06:29:53.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:53.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:53.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-07-26T06:29:53.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:29:53.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:53.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.130461 2021-07-26T06:29:53.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:29:53.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:29:53.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:29:53.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-07-26T06:29:53.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2021-07-26T06:29:53.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2021-07-26T06:29:53.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2021-07-26T06:29:53.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2021-07-26T06:29:53.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2021-07-26T06:29:53.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2021-07-26T06:29:53.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2021-07-26T06:29:53.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2021-07-26T06:29:53.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2021-07-26T06:29:53.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2021-07-26T06:29:53.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2021-07-26T06:29:53.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2021-07-26T06:29:53.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2021-07-26T06:29:53.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2021-07-26T06:29:53.353 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2021-07-26T06:29:53.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2021-07-26T06:29:53.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2021-07-26T06:29:53.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2021-07-26T06:29:53.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2021-07-26T06:29:53.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-07-26T06:29:53.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-07-26T06:29:53.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:53.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-07-26T06:29:53.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:53.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:53.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:53.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:53.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:53.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:53.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:53.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:53.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:53.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:53.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:53.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:53.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:53.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:53.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:53.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:53.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-07-26T06:29:53.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:53.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:53.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.130461 2021-07-26T06:29:53.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:53.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:53.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:29:53.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-config 0 2021-07-26T06:29:53.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-07-26T06:29:53.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:29:53.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-07-26T06:29:53.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:53.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:53.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:53.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:53.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:53.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:53.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:53.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:53.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:53.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:53.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:53.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:53.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:53.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:53.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:53.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:29:53.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-07-26T06:29:53.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:53.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.130461 2021-07-26T06:29:53.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.130461 2021-07-26T06:29:53.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:53.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:53.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:29:53.409 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:29:53.410 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:29:53.424 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2021-07-26T06:29:53.425 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2021-07-26T06:29:53.442 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:29:53.456 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:29:53.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:29:53.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:29:53.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:29:53.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:29:53.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:29:53.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:29:53.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:29:53.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:29:53.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-copy-from 2021-07-26T06:29:53.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2021-07-26T06:29:53.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2021-07-26T06:29:53.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2021-07-26T06:29:53.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2021-07-26T06:29:53.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2021-07-26T06:29:53.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2021-07-26T06:29:53.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=d94e2801-ed21-476d-9428-d28fb742f8d4 --auth-supported=none ' 2021-07-26T06:29:53.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2021-07-26T06:29:53.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2021-07-26T06:29:53.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:29:53.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2021-07-26T06:29:53.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2021-07-26T06:29:53.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2021-07-26T06:29:53.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2021-07-26T06:29:53.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2021-07-26T06:29:53.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-07-26T06:29:53.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:29:53.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-07-26T06:29:53.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:29:53.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:29:53.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:29:53.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:29:53.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:29:53.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:29:53.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:29:53.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:29:53.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:29:53.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:29:53.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:29:53.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:29:53.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:29:53.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:29:53.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:29:53.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:29:53.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:29:53.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-07-26T06:29:53.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:29:53.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:29:53.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.133943 2021-07-26T06:29:53.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:29:53.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:29:53.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2021-07-26T06:29:53.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:29:53.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:29:53.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.133943 2021-07-26T06:29:53.483 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:29:53.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:29:53.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:29:53.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:29:53.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2021-07-26T06:29:53.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2021-07-26T06:29:53.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2021-07-26T06:29:53.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2021-07-26T06:29:53.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2021-07-26T06:29:53.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:29:53.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:29:53.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:29:53.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2021-07-26T06:29:53.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2021-07-26T06:29:53.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:29:53.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:53.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:53.902 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:53.903 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:53.903 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:29:53.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.133943/$cluster-$name.asok' 2021-07-26T06:29:53.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:29:54.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:29:54.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:29:54.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:54.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:54.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:29:54.196 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:29:54.196 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:54.197 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:54.197 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:54.197 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:54.197 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:54.198 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:29:54.198 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.133943/ceph-mon.a.asok 2021-07-26T06:29:54.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:54.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.133943/ceph-mon.a.asok config get fsid 2021-07-26T06:29:54.282 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:29:54.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:29:54.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:29:54.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:29:54.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:29:54.291 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:29:54.291 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:29:54.291 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:29:54.292 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:29:54.292 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:54.292 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:29:54.292 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.133943/ceph-mon.a.asok 2021-07-26T06:29:54.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:29:54.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.133943/ceph-mon.a.asok config get mon_host 2021-07-26T06:29:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2021-07-26T06:29:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2021-07-26T06:29:54.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:29:54.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:29:54.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:29:54.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2021-07-26T06:29:54.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:29:55.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:29:55.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:55.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:55.284 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:55.284 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:55.285 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:29:55.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.133943/$cluster-$name.asok' 2021-07-26T06:29:55.286 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:55.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:29:55.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2021-07-26T06:29:55.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-07-26T06:29:55.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:29:55.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:29:55.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:29:55.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2021-07-26T06:29:55.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d94e2801-ed21-476d-9428-d28fb742f8d4 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-07-26T06:29:55.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:29:55.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:29:55.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:29:55.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2021-07-26T06:29:55.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2021-07-26T06:29:55.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:29:55.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:29:55.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:29:55.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-07-26T06:29:55.352 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:29:55.352 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:29:55.353 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:29:55.353 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:29:55.353 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:29:55.353 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:29:55.354 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.133943/$cluster-$name.asok' 2021-07-26T06:29:55.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' 2021-07-26T06:29:55.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:29:55.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:29:55.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:29:55.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-07-26T06:29:55.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-07-26T06:29:55.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:29:55.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:29:55.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:29:55.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:29:55.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:29:55.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2021-07-26T06:29:55.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:29:55.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03696d74-2799-4839-a453-ae336fe16af1 2021-07-26T06:29:55.359 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 03696d74-2799-4839-a453-ae336fe16af1 2021-07-26T06:29:55.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 03696d74-2799-4839-a453-ae336fe16af1' 2021-07-26T06:29:55.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:29:55.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjVv5gs2STFRAAt9DyNNH+AkecWZkJ3oYucA== 2021-07-26T06:29:55.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjVv5gs2STFRAAt9DyNNH+AkecWZkJ3oYucA=="}' 2021-07-26T06:29:55.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03696d74-2799-4839-a453-ae336fe16af1 -i td/osd-copy-from/0/new.json 2021-07-26T06:29:55.776 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:29:55.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2021-07-26T06:29:55.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d94e2801-ed21-476d-9428-d28fb742f8d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBjVv5gs2STFRAAt9DyNNH+AkecWZkJ3oYucA== --osd-uuid 03696d74-2799-4839-a453-ae336fe16af1 2021-07-26T06:29:55.821 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:55.802+0000 7f9971166f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:55.823 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:55.802+0000 7f9971166f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:55.825 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:55.806+0000 7f9971166f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:29:55.826 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:55.806+0000 7f9971166f40 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-07-26T06:29:55.826 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:55.806+0000 7f9971166f40 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-07-26T06:29:55.826 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:55.806+0000 7f9971166f40 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-07-26T06:29:55.907 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:29:55.886+0000 7f9971166f40 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2021-07-26T06:30:01.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2021-07-26T06:30:01.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:30:01.804 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:30:01.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:30:01.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:30:02.322 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:30:02.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:30:02.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d94e2801-ed21-476d-9428-d28fb742f8d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:02.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:30:02.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:30:02.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:30:02.350 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:02.330+0000 7f5753ef3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:02.352 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:02.334+0000 7f5753ef3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:02.353 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:02.334+0000 7f5753ef3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:02.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:30:02.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:02.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:30:02.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:02.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:02.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:02.692 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:30:02.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:02.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:02.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:02.890 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:02.870+0000 7f5753ef3f40 -1 Falling back to public interface 2021-07-26T06:30:03.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:04.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:04.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:04.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:04.083 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:30:04.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:04.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:04.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:05.238 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:05.218+0000 7f5753ef3f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:30:05.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:05.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:05.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:30:05.461 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:30:05.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:05.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:05.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:06.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:06.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:06.908 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:30:06.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:30:06.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:06.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:07.263 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/135047,v1:127.0.0.1:6803/135047] [v2:127.0.0.1:6804/135047,v1:127.0.0.1:6805/135047] exists,up 03696d74-2799-4839-a453-ae336fe16af1 2021-07-26T06:30:07.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:07.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:07.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:07.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2021-07-26T06:30:07.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2021-07-26T06:30:07.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:30:07.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:30:07.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:30:07.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2021-07-26T06:30:07.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d94e2801-ed21-476d-9428-d28fb742f8d4 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-07-26T06:30:07.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:30:07.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:30:07.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:30:07.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2021-07-26T06:30:07.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2021-07-26T06:30:07.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:30:07.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:30:07.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:30:07.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-07-26T06:30:07.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:30:07.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:07.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:07.272 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:07.272 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:07.272 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:30:07.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.133943/$cluster-$name.asok' 2021-07-26T06:30:07.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' 2021-07-26T06:30:07.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:30:07.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:30:07.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:30:07.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-07-26T06:30:07.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-07-26T06:30:07.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:30:07.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:30:07.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:07.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:30:07.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:30:07.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2021-07-26T06:30:07.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:30:07.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=914b06af-c7bb-4cec-a43b-99a63b04979c 2021-07-26T06:30:07.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 914b06af-c7bb-4cec-a43b-99a63b04979c' 2021-07-26T06:30:07.277 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 914b06af-c7bb-4cec-a43b-99a63b04979c 2021-07-26T06:30:07.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:30:07.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvVv5gqPMqEBAAQhuDo2/5f3oDKKpJplE6Iw== 2021-07-26T06:30:07.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvVv5gqPMqEBAAQhuDo2/5f3oDKKpJplE6Iw=="}' 2021-07-26T06:30:07.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 914b06af-c7bb-4cec-a43b-99a63b04979c -i td/osd-copy-from/1/new.json 2021-07-26T06:30:07.859 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:30:07.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2021-07-26T06:30:07.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d94e2801-ed21-476d-9428-d28fb742f8d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBvVv5gqPMqEBAAQhuDo2/5f3oDKKpJplE6Iw== --osd-uuid 914b06af-c7bb-4cec-a43b-99a63b04979c 2021-07-26T06:30:07.909 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:07.890+0000 7f3f52241f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:07.912 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:07.894+0000 7f3f52241f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:07.914 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:07.894+0000 7f3f52241f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:07.894+0000 7f3f52241f40 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-07-26T06:30:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:07.894+0000 7f3f52241f40 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-07-26T06:30:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:07.894+0000 7f3f52241f40 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-07-26T06:30:08.056 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:08.038+0000 7f3f52241f40 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2021-07-26T06:30:14.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2021-07-26T06:30:14.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:30:14.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:30:14.439 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:30:14.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:30:14.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:30:14.927 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:30:14.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d94e2801-ed21-476d-9428-d28fb742f8d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.133943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:14.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:30:14.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:30:14.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:30:14.954 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:14.934+0000 7fa2f833cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:14.957 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:14.938+0000 7fa2f833cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:14.958 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:14.938+0000 7fa2f833cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:15.242 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:15.222+0000 7fa2f833cf40 -1 Falling back to public interface 2021-07-26T06:30:15.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:30:15.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:15.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:30:15.271 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:30:15.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:15.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:15.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:15.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:15.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:15.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:30:15.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:16.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:16.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:16.731 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:30:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:16.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:30:17.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:18.078 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:18.058+0000 7fa2f833cf40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:30:18.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:18.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:18.232 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:30:18.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:30:18.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:18.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:30:18.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:19.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:19.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:19.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:30:19.750 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:30:19.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:19.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:30:20.107 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/136453,v1:127.0.0.1:6811/136453] [v2:127.0.0.1:6812/136453,v1:127.0.0.1:6813/136453] exists,up 914b06af-c7bb-4cec-a43b-99a63b04979c 2021-07-26T06:30:20.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:20.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:20.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:20.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2021-07-26T06:30:20.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:30:20.552 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' does not exist 2021-07-26T06:30:20.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:30:20.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:30:21.426 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' already exists 2021-07-26T06:30:21.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:30:22.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:30:24.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2021-07-26T06:30:24.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2021-07-26T06:30:24.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2021-07-26T06:30:24.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2021-07-26T06:30:24.494 INFO:tasks.workunit.client.0.smithi094.stdout:rbd/foo2 mtime 2021-07-26T06:30:24.000000+0000, size 670128 2021-07-26T06:30:24.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2021-07-26T06:30:24.630 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0: {} 2021-07-26T06:30:24.663 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1: {} 2021-07-26T06:30:24.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2021-07-26T06:30:24.755 INFO:tasks.workunit.client.0.smithi094.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2021-07-26T06:30:24.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2021-07-26T06:30:24.794 INFO:tasks.workunit.client.0.smithi094.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2021-07-26T06:30:24.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2021-07-26T06:30:24.933 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0: {} 2021-07-26T06:30:24.965 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1: {} 2021-07-26T06:30:24.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2021-07-26T06:30:25.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2021-07-26T06:30:25.272 INFO:tasks.workunit.client.0.smithi094.stdout:rbd/foo3 mtime 2021-07-26T06:30:24.000000+0000, size 670128 2021-07-26T06:30:25.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2021-07-26T06:30:25.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-07-26T06:30:25.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:30:25.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-07-26T06:30:25.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:30:25.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:30:25.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:30:25.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:30:25.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:30:25.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:30:25.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:30:25.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:30:25.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:30:25.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:30:25.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:30:25.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:30:25.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:30:25.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:30:25.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:30:25.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:30:25.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:30:25.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:30:25.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-07-26T06:30:25.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:30:25.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:25.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:30:25.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.133943 2021-07-26T06:30:25.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:30:25.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:30:25.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:30:25.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-copy-from 0 2021-07-26T06:30:25.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-07-26T06:30:25.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:30:25.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-07-26T06:30:25.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:30:25.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:30:25.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:30:25.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:30:25.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:30:25.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:30:25.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:30:25.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:30:25.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:30:25.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:30:25.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:30:25.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:30:25.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:30:25.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:30:25.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:30:25.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:30:25.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:30:25.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:30:25.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-07-26T06:30:25.431 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:30:25.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:25.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.133943 2021-07-26T06:30:25.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.133943 2021-07-26T06:30:25.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:30:25.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:30:25.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:30:25.435 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:30:25.435 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:30:25.490 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2021-07-26T06:30:25.491 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2021-07-26T06:30:25.507 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:30:25.521 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:30:25.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:30:25.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:30:25.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:30:25.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:30:25.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:30:25.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:30:25.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:30:25.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:30:25.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-dup 2021-07-26T06:30:25.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2021-07-26T06:30:25.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2021-07-26T06:30:25.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-07-26T06:30:25.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-07-26T06:30:25.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2021-07-26T06:30:25.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2021-07-26T06:30:25.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none ' 2021-07-26T06:30:25.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-07-26T06:30:25.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2021-07-26T06:30:25.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-07-26T06:30:25.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2021-07-26T06:30:25.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:30:25.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2021-07-26T06:30:25.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2021-07-26T06:30:25.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2021-07-26T06:30:25.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2021-07-26T06:30:25.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2021-07-26T06:30:25.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-07-26T06:30:25.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:30:25.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-07-26T06:30:25.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:30:25.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:30:25.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:30:25.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:30:25.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:30:25.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:30:25.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:30:25.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:30:25.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:30:25.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:30:25.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:30:25.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:30:25.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:30:25.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:30:25.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:30:25.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:30:25.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:30:25.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:30:25.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-07-26T06:30:25.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:30:25.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:25.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:25.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137330 2021-07-26T06:30:25.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:30:25.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:30:25.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2021-07-26T06:30:25.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:30:25.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:25.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:25.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.137330 2021-07-26T06:30:25.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:30:25.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:30:25.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:30:25.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:30:25.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2021-07-26T06:30:25.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2021-07-26T06:30:25.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2021-07-26T06:30:25.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2021-07-26T06:30:25.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2021-07-26T06:30:25.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2021-07-26T06:30:25.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2021-07-26T06:30:25.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2021-07-26T06:30:25.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:30:25.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:30:25.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:30:25.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2021-07-26T06:30:25.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2021-07-26T06:30:25.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:30:25.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:25.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:25.988 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:25.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:25.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:25.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:25.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:30:26.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:30:26.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:30:26.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:30:26.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:30:26.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:30:26.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:30:26.306 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:30:26.306 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:30:26.306 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:30:26.307 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:30:26.307 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:26.307 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:26.307 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137330/ceph-mon.a.asok 2021-07-26T06:30:26.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:30:26.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.137330/ceph-mon.a.asok config get fsid 2021-07-26T06:30:26.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:30:26.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:30:26.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:30:26.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:30:26.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:30:26.398 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:30:26.399 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:30:26.399 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:30:26.400 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:30:26.400 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:26.400 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:26.401 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.137330/ceph-mon.a.asok 2021-07-26T06:30:26.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:30:26.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.137330/ceph-mon.a.asok config get mon_host 2021-07-26T06:30:26.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2021-07-26T06:30:26.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-dup 2021-07-26T06:30:26.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:30:26.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:30:26.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:30:26.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-dup/x 2021-07-26T06:30:26.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:30:27.408 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:30:27.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:27.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:27.409 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:27.410 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:27.410 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:27.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:27.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:30:27.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:30:27.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2021-07-26T06:30:27.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2021-07-26T06:30:27.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:30:27.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2021-07-26T06:30:27.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:30:27.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/0 2021-07-26T06:30:27.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-07-26T06:30:27.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:30:27.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:30:27.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:30:27.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2021-07-26T06:30:27.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-07-26T06:30:27.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:30:27.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:30:27.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:30:27.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-07-26T06:30:27.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:30:27.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:27.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:27.448 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:27.448 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:27.448 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:27.449 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:27.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:27.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:30:27.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:30:27.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:30:27.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-07-26T06:30:27.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-07-26T06:30:27.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:30:27.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:30:27.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:27.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:30:27.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:30:27.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-dup/0 2021-07-26T06:30:27.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:30:27.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=7b34a481-6534-400c-8f9f-c017a4a9ebd1 2021-07-26T06:30:27.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd 7b34a481-6534-400c-8f9f-c017a4a9ebd1' 2021-07-26T06:30:27.453 INFO:tasks.workunit.client.0.smithi094.stdout:add osd 7b34a481-6534-400c-8f9f-c017a4a9ebd1 2021-07-26T06:30:27.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:30:27.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQCDVv5gtqlAGxAARQ83qm0S913deugzrT68kg== 2021-07-26T06:30:27.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQCDVv5gtqlAGxAARQ83qm0S913deugzrT68kg=="}' 2021-07-26T06:30:27.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 7b34a481-6534-400c-8f9f-c017a4a9ebd1 -i td/osd-dup/0/new.json 2021-07-26T06:30:27.838 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:30:27.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-dup/0/new.json 2021-07-26T06:30:27.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 0 --fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCDVv5gtqlAGxAARQ83qm0S913deugzrT68kg== --osd-uuid 7b34a481-6534-400c-8f9f-c017a4a9ebd1 --osd-objectstore=filestore 2021-07-26T06:30:27.882 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:27.863+0000 7f4d689e9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:27.885 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:27.867+0000 7f4d689e9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:27.887 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:27.867+0000 7f4d689e9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:28.413 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:28.395+0000 7f4d689e9f40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:29.111 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:29.091+0000 7f4d689e9f40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:29.111 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:29.091+0000 7f4d689e9f40 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:30:29.112 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:29.091+0000 7f4d689e9f40 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:30:29.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2021-07-26T06:30:29.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:30:29.696 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:30:29.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd0 key to auth repository 2021-07-26T06:30:29.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:30:29.871 INFO:tasks.workunit.client.0.smithi094.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-07-26T06:30:29.873 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:30:29.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.0 2021-07-26T06:30:29.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:30:29.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 0 --fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:29.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:30:29.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:30:29.901 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:29.883+0000 7f466c4f6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:29.903 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:29.887+0000 7f466c4f6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:29.904 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:29.887+0000 7f466c4f6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:29.911 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:29.895+0000 7f466c4f6f40 -1 Falling back to public interface 2021-07-26T06:30:30.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 0 2021-07-26T06:30:30.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:30.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:30:30.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:30.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:30.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:30.360 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:30:30.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:30.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:30.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:30.592 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:30.575+0000 7f466c4f6f40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:30.598 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:30.579+0000 7f466c4f6f40 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:30:30.598 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:30.579+0000 7f466c4f6f40 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:30:30.606 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:30.587+0000 7f466c4f6f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:30:30.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:31.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:31.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:31.622 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:30:31.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:31.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:31.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:30:32.013 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/137650,v1:127.0.0.1:6801/137650] [v2:127.0.0.1:6802/137650,v1:127.0.0.1:6803/137650] exists,up 7b34a481-6534-400c-8f9f-c017a4a9ebd1 2021-07-26T06:30:32.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:32.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:32.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:32.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2021-07-26T06:30:32.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=137650 2021-07-26T06:30:32.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2021-07-26T06:30:32.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2021-07-26T06:30:32.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:30:32.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2021-07-26T06:30:32.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:30:32.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/1 2021-07-26T06:30:32.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-07-26T06:30:32.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:30:32.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:30:32.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:30:32.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2021-07-26T06:30:32.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2021-07-26T06:30:32.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:30:32.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:30:32.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:30:32.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-07-26T06:30:32.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:30:32.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:32.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:32.034 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:32.034 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:32.034 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:32.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:32.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:32.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:30:32.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:30:32.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:30:32.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-07-26T06:30:32.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-07-26T06:30:32.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:30:32.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:30:32.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:32.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:30:32.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:30:32.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-dup/1 2021-07-26T06:30:32.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:30:32.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=b6136d45-fa8f-441d-aa9c-2cbd376c4d06 2021-07-26T06:30:32.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd b6136d45-fa8f-441d-aa9c-2cbd376c4d06' 2021-07-26T06:30:32.042 INFO:tasks.workunit.client.0.smithi094.stdout:add osd b6136d45-fa8f-441d-aa9c-2cbd376c4d06 2021-07-26T06:30:32.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:30:32.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQCIVv5ghZKmARAAwLq81h39n0YwJgDUf4GhNQ== 2021-07-26T06:30:32.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQCIVv5ghZKmARAAwLq81h39n0YwJgDUf4GhNQ=="}' 2021-07-26T06:30:32.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new b6136d45-fa8f-441d-aa9c-2cbd376c4d06 -i td/osd-dup/1/new.json 2021-07-26T06:30:32.577 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:30:32.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-dup/1/new.json 2021-07-26T06:30:32.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 1 --fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCIVv5ghZKmARAAwLq81h39n0YwJgDUf4GhNQ== --osd-uuid b6136d45-fa8f-441d-aa9c-2cbd376c4d06 --osd-objectstore=filestore 2021-07-26T06:30:32.622 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:32.603+0000 7f845b1aff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:32.628 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:32.611+0000 7f845b1aff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:32.630 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:32.611+0000 7f845b1aff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:33.105 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:33.087+0000 7f845b1aff40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:33.936 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:33.919+0000 7f845b1aff40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:33.936 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:33.919+0000 7f845b1aff40 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:30:33.936 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:33.919+0000 7f845b1aff40 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:30:34.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2021-07-26T06:30:34.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:30:34.496 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:30:34.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd1 key to auth repository 2021-07-26T06:30:34.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:30:34.819 INFO:tasks.workunit.client.0.smithi094.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-07-26T06:30:34.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.1 2021-07-26T06:30:34.823 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:30:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 1 --fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:30:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:30:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:30:34.846 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:34.827+0000 7f8c526ebf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:34.849 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:34.831+0000 7f8c526ebf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:34.851 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:34.831+0000 7f8c526ebf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:34.851 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:34.835+0000 7f8c526ebf40 -1 Falling back to public interface 2021-07-26T06:30:35.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 1 2021-07-26T06:30:35.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:35.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:30:35.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:35.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:35.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:35.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:35.329 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:30:35.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:35.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:30:35.493 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:35.475+0000 7f8c526ebf40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:35.494 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:35.475+0000 7f8c526ebf40 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:30:35.494 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:35.475+0000 7f8c526ebf40 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:30:35.502 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:35.483+0000 7f8c526ebf40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:30:35.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:36.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:36.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:36.769 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:30:36.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:36.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:36.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:30:37.136 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/138012,v1:127.0.0.1:6811/138012] [v2:127.0.0.1:6812/138012,v1:127.0.0.1:6813/138012] exists,up b6136d45-fa8f-441d-aa9c-2cbd376c4d06 2021-07-26T06:30:37.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:37.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:37.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:37.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2021-07-26T06:30:37.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2021-07-26T06:30:37.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-07-26T06:30:37.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2021-07-26T06:30:37.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2021-07-26T06:30:37.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/2 2021-07-26T06:30:37.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-07-26T06:30:37.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:30:37.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:30:37.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:30:37.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2021-07-26T06:30:37.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2021-07-26T06:30:37.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:30:37.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --chdir=' 2021-07-26T06:30:37.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+= 2021-07-26T06:30:37.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-07-26T06:30:37.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: get_asok_path 2021-07-26T06:30:37.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:30:37.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:30:37.145 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:30:37.145 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:30:37.145 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:30:37.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:37.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:30:37.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-07-26T06:30:37.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-07-26T06:30:37.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-07-26T06:30:37.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-07-26T06:30:37.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-07-26T06:30:37.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:30:37.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:30:37.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:37.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+=' ' 2021-07-26T06:30:37.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: ceph_args+= 2021-07-26T06:30:37.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: mkdir -p td/osd-dup/2 2021-07-26T06:30:37.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: uuidgen 2021-07-26T06:30:37.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: local uuid=7fabbeca-ded6-44de-8985-9038d0d80cab 2021-07-26T06:30:37.149 INFO:tasks.workunit.client.0.smithi094.stdout:add osd 7fabbeca-ded6-44de-8985-9038d0d80cab 2021-07-26T06:30:37.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: echo 'add osd 7fabbeca-ded6-44de-8985-9038d0d80cab' 2021-07-26T06:30:37.150 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: ceph-authtool --gen-print-key 2021-07-26T06:30:37.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: OSD_SECRET=AQCNVv5glTfRCBAATfpTm78bMmswSqe+MdHvrg== 2021-07-26T06:30:37.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: echo '{"cephx_secret": "AQCNVv5glTfRCBAATfpTm78bMmswSqe+MdHvrg=="}' 2021-07-26T06:30:37.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: ceph osd new 7fabbeca-ded6-44de-8985-9038d0d80cab -i td/osd-dup/2/new.json 2021-07-26T06:30:37.861 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:30:37.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: rm td/osd-dup/2/new.json 2021-07-26T06:30:37.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: ceph-osd -i 2 --fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCNVv5glTfRCBAATfpTm78bMmswSqe+MdHvrg== --osd-uuid 7fabbeca-ded6-44de-8985-9038d0d80cab --osd-objectstore=filestore 2021-07-26T06:30:37.916 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:37.899+0000 7fded30a7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:38.005 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:37.987+0000 7fded30a7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:38.007 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:37.991+0000 7fded30a7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:38.402 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:38.383+0000 7fded30a7f40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:39.159 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:39.139+0000 7fded30a7f40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:39.159 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:39.139+0000 7fded30a7f40 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:30:39.159 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:39.139+0000 7fded30a7f40 -1 journal do_read_entry(4096): bad header magic 2021-07-26T06:30:39.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2021-07-26T06:30:39.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:727: run_osd_filestore: cat 2021-07-26T06:30:39.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: echo adding osd2 key to auth repository 2021-07-26T06:30:39.923 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:30:39.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:30:40.250 INFO:tasks.workunit.client.0.smithi094.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-07-26T06:30:40.253 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:30:40.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: echo start osd.2 2021-07-26T06:30:40.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: ceph osd dump --format=json 2021-07-26T06:30:40.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph-osd -i 2 --fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:30:40.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: jq '.flags_set[]' 2021-07-26T06:30:40.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: grep -q '"noup"' 2021-07-26T06:30:40.279 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:40.259+0000 7ff19c9a4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:40.281 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:40.263+0000 7ff19c9a4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:40.283 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:40.267+0000 7ff19c9a4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:30:40.284 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:40.267+0000 7ff19c9a4f40 -1 Falling back to public interface 2021-07-26T06:30:40.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: run_osd_filestore: wait_for_osd up 2 2021-07-26T06:30:40.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:30:40.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:30:40.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:30:40.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:30:40.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:40.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:30:40.850 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:30:40.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:30:40.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:41.227 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:41.211+0000 7ff19c9a4f40 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-07-26T06:30:41.231 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:41.215+0000 7ff19c9a4f40 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:30:41.231 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:41.215+0000 7ff19c9a4f40 -1 journal do_read_entry(8192): bad header magic 2021-07-26T06:30:41.240 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:30:41.223+0000 7ff19c9a4f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:30:41.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:30:42.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:30:42.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:30:42.293 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:30:42.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:30:42.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:30:42.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:30:42.720 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/138299,v1:127.0.0.1:6819/138299] [v2:127.0.0.1:6820/138299,v1:127.0.0.1:6821/138299] exists,up 7fabbeca-ded6-44de-8985-9038d0d80cab 2021-07-26T06:30:42.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:30:42.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:30:42.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:30:42.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2021-07-26T06:30:47.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2021-07-26T06:30:47.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2021-07-26T06:30:48.271 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'foo' already exists 2021-07-26T06:30:48.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:30:49.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2021-07-26T06:30:49.328 INFO:tasks.workunit.client.0.smithi094.stdout:hints = 1 2021-07-26T06:30:49.328 INFO:tasks.workunit.client.0.smithi094.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2021-07-26T06:30:49.328 INFO:tasks.workunit.client.0.smithi094.stdout:Object prefix: benchmark_data_smithi094_138483 2021-07-26T06:30:49.328 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:30:49.329 INFO:tasks.workunit.client.0.smithi094.stdout: 0 0 0 0 0 0 - 0 2021-07-26T06:30:50.328 INFO:tasks.workunit.client.0.smithi094.stdout: 1 16 16 0 0 0 - 0 2021-07-26T06:30:51.328 INFO:tasks.workunit.client.0.smithi094.stdout: 2 16 16 0 0 0 - 0 2021-07-26T06:30:52.328 INFO:tasks.workunit.client.0.smithi094.stdout: 3 16 16 0 0 0 - 0 2021-07-26T06:30:53.328 INFO:tasks.workunit.client.0.smithi094.stdout: 4 16 21 5 0.00488212 0.00488281 3.995 3.94025 2021-07-26T06:30:54.328 INFO:tasks.workunit.client.0.smithi094.stdout: 5 16 21 5 0.00390573 0 - 3.94025 2021-07-26T06:30:55.328 INFO:tasks.workunit.client.0.smithi094.stdout: 6 16 21 5 0.00325476 0 - 3.94025 2021-07-26T06:30:56.328 INFO:tasks.workunit.client.0.smithi094.stdout: 7 16 70 54 0.0301298 0.0638021 0.120455 2.06644 2021-07-26T06:30:57.329 INFO:tasks.workunit.client.0.smithi094.stdout: 8 16 134 118 0.0576092 0.25 0.3693 1.05494 2021-07-26T06:30:58.329 INFO:tasks.workunit.client.0.smithi094.stdout: 9 16 215 199 0.0863593 0.316406 0.140725 0.712926 2021-07-26T06:30:59.329 INFO:tasks.workunit.client.0.smithi094.stdout: 10 16 327 311 0.121467 0.4375 0.146025 0.509723 2021-07-26T06:31:00.329 INFO:tasks.workunit.client.0.smithi094.stdout:Total time run: 10.0656 2021-07-26T06:31:00.330 INFO:tasks.workunit.client.0.smithi094.stdout:Total writes made: 327 2021-07-26T06:31:00.330 INFO:tasks.workunit.client.0.smithi094.stdout:Write size: 4096 2021-07-26T06:31:00.330 INFO:tasks.workunit.client.0.smithi094.stdout:Object size: 4096 2021-07-26T06:31:00.330 INFO:tasks.workunit.client.0.smithi094.stdout:Bandwidth (MB/sec): 0.126901 2021-07-26T06:31:00.331 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Bandwidth: 0.16434 2021-07-26T06:31:00.331 INFO:tasks.workunit.client.0.smithi094.stdout:Max bandwidth (MB/sec): 0.4375 2021-07-26T06:31:00.331 INFO:tasks.workunit.client.0.smithi094.stdout:Min bandwidth (MB/sec): 0 2021-07-26T06:31:00.331 INFO:tasks.workunit.client.0.smithi094.stdout:Average IOPS: 32 2021-07-26T06:31:00.331 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev IOPS: 42.0983 2021-07-26T06:31:00.332 INFO:tasks.workunit.client.0.smithi094.stdout:Max IOPS: 112 2021-07-26T06:31:00.332 INFO:tasks.workunit.client.0.smithi094.stdout:Min IOPS: 0 2021-07-26T06:31:00.332 INFO:tasks.workunit.client.0.smithi094.stdout:Average Latency(s): 0.490486 2021-07-26T06:31:00.332 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Latency(s): 1.29466 2021-07-26T06:31:00.333 INFO:tasks.workunit.client.0.smithi094.stdout:Max latency(s): 6.73863 2021-07-26T06:31:00.333 INFO:tasks.workunit.client.0.smithi094.stdout:Min latency(s): 0.104186 2021-07-26T06:31:00.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 137650 2021-07-26T06:31:00.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2021-07-26T06:31:01.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 137650 2021-07-26T06:31:01.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (137650) - No such process 2021-07-26T06:31:01.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2021-07-26T06:31:01.782 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T06:31:01.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2021-07-26T06:31:01.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2021-07-26T06:31:01.794 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2021-07-26T06:31:01.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=7b34a481-6534-400c-8f9f-c017a4a9ebd1 2021-07-26T06:31:01.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid 7b34a481-6534-400c-8f9f-c017a4a9ebd1' 2021-07-26T06:31:01.796 INFO:tasks.workunit.client.0.smithi094.stdout:osd fsid 7b34a481-6534-400c-8f9f-c017a4a9ebd1 2021-07-26T06:31:01.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-07-26T06:31:01.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2021-07-26T06:31:01.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid 7b34a481-6534-400c-8f9f-c017a4a9ebd1 --op mkfs --no-mon-config 2021-07-26T06:31:07.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2021-07-26T06:31:07.103 INFO:tasks.workunit.client.0.smithi094.stdout:dup from filestore: td/osd-dup/0.old 2021-07-26T06:31:07.103 INFO:tasks.workunit.client.0.smithi094.stdout: to bluestore: td/osd-dup/0 2021-07-26T06:31:08.491 INFO:tasks.workunit.client.0.smithi094.stdout:fsid 7b34a481-6534-400c-8f9f-c017a4a9ebd1 2021-07-26T06:31:08.492 INFO:tasks.workunit.client.0.smithi094.stdout:17 collections 2021-07-26T06:31:08.492 INFO:tasks.workunit.client.0.smithi094.stdout:1/17 1.e_head 2021-07-26T06:31:08.534 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 16 objects, 61440 bytes, 21 keys 2021-07-26T06:31:08.534 INFO:tasks.workunit.client.0.smithi094.stdout:2/17 1.c_head 2021-07-26T06:31:08.671 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 18 objects, 69632 bytes, 23 keys 2021-07-26T06:31:08.672 INFO:tasks.workunit.client.0.smithi094.stdout:3/17 1.b_head 2021-07-26T06:31:08.769 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 26 objects, 102400 bytes, 31 keys 2021-07-26T06:31:08.769 INFO:tasks.workunit.client.0.smithi094.stdout:4/17 1.d_head 2021-07-26T06:31:08.837 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 25 objects, 94232 bytes, 30 keys 2021-07-26T06:31:08.837 INFO:tasks.workunit.client.0.smithi094.stdout:5/17 1.3_head 2021-07-26T06:31:08.971 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 30 objects, 118784 bytes, 35 keys 2021-07-26T06:31:08.971 INFO:tasks.workunit.client.0.smithi094.stdout:6/17 1.9_head 2021-07-26T06:31:09.030 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 18 objects, 69632 bytes, 23 keys 2021-07-26T06:31:09.030 INFO:tasks.workunit.client.0.smithi094.stdout:7/17 1.2_head 2021-07-26T06:31:09.036 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 21 objects, 81920 bytes, 26 keys 2021-07-26T06:31:09.036 INFO:tasks.workunit.client.0.smithi094.stdout:8/17 1.f_head 2021-07-26T06:31:09.115 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2021-07-26T06:31:09.115 INFO:tasks.workunit.client.0.smithi094.stdout:9/17 1.8_head 2021-07-26T06:31:09.158 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 18 objects, 69632 bytes, 23 keys 2021-07-26T06:31:09.158 INFO:tasks.workunit.client.0.smithi094.stdout:10/17 1.a_head 2021-07-26T06:31:09.284 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 21 objects, 81920 bytes, 26 keys 2021-07-26T06:31:09.284 INFO:tasks.workunit.client.0.smithi094.stdout:11/17 1.5_head 2021-07-26T06:31:09.294 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 18 objects, 69632 bytes, 23 keys 2021-07-26T06:31:09.295 INFO:tasks.workunit.client.0.smithi094.stdout:12/17 1.4_head 2021-07-26T06:31:09.364 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 25 objects, 98304 bytes, 30 keys 2021-07-26T06:31:09.364 INFO:tasks.workunit.client.0.smithi094.stdout:13/17 1.1_head 2021-07-26T06:31:09.403 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 18 objects, 69632 bytes, 23 keys 2021-07-26T06:31:09.403 INFO:tasks.workunit.client.0.smithi094.stdout:14/17 1.7_head 2021-07-26T06:31:09.411 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 19 objects, 73728 bytes, 24 keys 2021-07-26T06:31:09.411 INFO:tasks.workunit.client.0.smithi094.stdout:15/17 1.0_head 2021-07-26T06:31:09.444 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 16 objects, 61440 bytes, 21 keys 2021-07-26T06:31:09.444 INFO:tasks.workunit.client.0.smithi094.stdout:16/17 1.6_head 2021-07-26T06:31:09.653 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 27 objects, 106496 bytes, 32 keys 2021-07-26T06:31:09.654 INFO:tasks.workunit.client.0.smithi094.stdout:17/17 meta 2021-07-26T06:31:09.663 INFO:tasks.workunit.client.0.smithi094.stdout: 0 objects, 0 bytes, 0 keys 46 objects, 47243 bytes, 0 keys 2021-07-26T06:31:09.663 INFO:tasks.workunit.client.0.smithi094.stdout:keyring 2021-07-26T06:31:09.663 INFO:tasks.workunit.client.0.smithi094.stdout:duping osd metadata 2021-07-26T06:31:10.080 INFO:tasks.workunit.client.0.smithi094.stdout:done. 2021-07-26T06:31:10.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-07-26T06:31:10.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2021-07-26T06:31:10.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-dup 2021-07-26T06:31:10.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:31:10.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:31:10.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:31:10.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-dup/0 2021-07-26T06:31:10.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-07-26T06:31:10.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:31:10.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:31:10.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:31:10.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2021-07-26T06:31:10.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-07-26T06:31:10.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:31:10.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:31:10.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:31:10.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2021-07-26T06:31:10.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:31:10.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:10.946 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:10.948 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:10.948 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:10.948 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:31:10.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:31:10.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' 2021-07-26T06:31:10.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:31:10.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-07-26T06:31:10.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-07-26T06:31:10.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:31:10.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:31:10.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:10.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:31:10.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:31:10.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-dup/0 2021-07-26T06:31:10.951 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:31:10.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:31:10.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=af486139-5033-4a9e-8549-37ed8134b38c --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.137330/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:10.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-dup/0/whoami 2021-07-26T06:31:10.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:31:10.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:31:10.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:31:10.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:31:10.978 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:10.958+0000 7f9a85385f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:10.981 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:10.962+0000 7f9a85385f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:10.983 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:10.966+0000 7f9a85385f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:11.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:31:11.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:31:11.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:31:11.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:31:11.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:31:11.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:11.307 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:31:11.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:31:11.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:11.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:11.521 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:11.502+0000 7f9a85385f40 -1 Falling back to public interface 2021-07-26T06:31:11.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:12.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:12.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:12.665 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:31:12.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:31:12.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:12.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:13.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:14.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:14.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:14.020 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:31:14.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:31:14.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:14.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:14.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:14.633 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:14.614+0000 7f9a85385f40 -1 osd.0 21 log_to_monitors {default=true} 2021-07-26T06:31:15.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:15.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:15.391 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:31:15.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:31:15.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:15.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:15.938 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 26 up_thru 17 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/139357,v1:127.0.0.1:6801/139357] [v2:127.0.0.1:6802/139357,v1:127.0.0.1:6803/139357] exists,up 7b34a481-6534-400c-8f9f-c017a4a9ebd1 2021-07-26T06:31:15.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:31:15.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:31:15.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:31:15.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2021-07-26T06:31:15.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2021-07-26T06:31:16.442 INFO:tasks.workunit.client.0.smithi094.stdout:3 osds: 3 up (since 1.46545s), 3 in (since 38s); epoch: e28 2021-07-26T06:31:16.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2021-07-26T06:31:16.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2021-07-26T06:31:17.025 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_access_mode": "file", 2021-07-26T06:31:17.026 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_block_size": "4096", 2021-07-26T06:31:17.026 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_devices": "sda", 2021-07-26T06:31:17.027 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-07-26T06:31:17.027 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2021-07-26T06:31:17.028 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_rotational": "1", 2021-07-26T06:31:17.028 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_size": "107374182400", 2021-07-26T06:31:17.029 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_support_discard": "0", 2021-07-26T06:31:17.029 INFO:tasks.workunit.client.0.smithi094.stdout: "bluestore_bdev_type": "hdd", 2021-07-26T06:31:17.030 INFO:tasks.workunit.client.0.smithi094.stdout: "osd_objectstore": "bluestore", 2021-07-26T06:31:17.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2021-07-26T06:31:17.388 INFO:tasks.workunit.client.0.smithi094.stderr:instructed osd(s) 0 to scrub 2021-07-26T06:31:17.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2021-07-26T06:31:32.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2021-07-26T06:31:32.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:31:32.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:31:32.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:31:32.807 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:31:32.807 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:31:32.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:31:32.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:32.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:31:32.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149702 2021-07-26T06:31:32.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149702 2021-07-26T06:31:32.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149702' 2021-07-26T06:31:32.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:32.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:31:33.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672974 2021-07-26T06:31:33.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672974 2021-07-26T06:31:33.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149702 1-42949672974' 2021-07-26T06:31:33.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:33.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:31:33.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509453 2021-07-26T06:31:33.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509453 2021-07-26T06:31:33.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149702 1-42949672974 2-64424509453' 2021-07-26T06:31:33.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:33.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149702 2021-07-26T06:31:33.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:33.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:31:33.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149702 2021-07-26T06:31:33.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:33.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149702 2021-07-26T06:31:33.263 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 111669149702 2021-07-26T06:31:33.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2021-07-26T06:31:33.265 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:33.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149700 -lt 111669149702 2021-07-26T06:31:33.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:31:34.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:31:34.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:34.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149702 -lt 111669149702 2021-07-26T06:31:34.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:34.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672974 2021-07-26T06:31:34.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:34.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:31:34.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672974 2021-07-26T06:31:34.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:34.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672974 2021-07-26T06:31:34.983 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672974 2021-07-26T06:31:34.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-07-26T06:31:34.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:31:35.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672974 2021-07-26T06:31:35.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:35.346 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509453 2021-07-26T06:31:35.346 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:35.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:31:35.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509453 2021-07-26T06:31:35.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:35.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509453 2021-07-26T06:31:35.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-07-26T06:31:35.352 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509453 2021-07-26T06:31:35.352 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:31:35.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509453 -lt 64424509453 2021-07-26T06:31:35.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2021-07-26T06:31:35.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:31:35.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:31:35.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:31:35.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:31:35.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:31:35.738 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:31:35.738 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:31:35.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:31:35.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:31:35.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:31:35.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:31:35.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:31:35.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:31:35.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:31:35.937 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:31:36.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:31:36.303 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:31:36.303 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:31:36.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:31:36.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:36.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:31:36.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149705 2021-07-26T06:31:36.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149705 2021-07-26T06:31:36.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149705' 2021-07-26T06:31:36.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:36.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:31:36.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672976 2021-07-26T06:31:36.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672976 2021-07-26T06:31:36.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149705 1-42949672976' 2021-07-26T06:31:36.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:31:36.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:31:36.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509455 2021-07-26T06:31:36.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509455 2021-07-26T06:31:36.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149705 1-42949672976 2-64424509455' 2021-07-26T06:31:36.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:36.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149705 2021-07-26T06:31:36.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:36.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:31:36.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149705 2021-07-26T06:31:36.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:36.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149705 2021-07-26T06:31:36.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2021-07-26T06:31:36.727 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 111669149705 2021-07-26T06:31:36.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:37.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149703 -lt 111669149705 2021-07-26T06:31:37.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:31:38.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:31:38.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:31:38.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149705 -lt 111669149705 2021-07-26T06:31:38.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:38.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672976 2021-07-26T06:31:38.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:38.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:31:38.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672976 2021-07-26T06:31:38.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:38.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672976 2021-07-26T06:31:38.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-07-26T06:31:38.503 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672976 2021-07-26T06:31:38.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:31:38.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672976 -lt 42949672976 2021-07-26T06:31:38.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:31:38.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509455 2021-07-26T06:31:38.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:31:38.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:31:38.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509455 2021-07-26T06:31:38.881 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:31:38.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509455 2021-07-26T06:31:38.883 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509455 2021-07-26T06:31:38.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-07-26T06:31:38.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:31:39.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509455 -lt 64424509455 2021-07-26T06:31:39.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:31:39.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:39.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:39.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 16 == 0 2021-07-26T06:31:39.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:31:39.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:31:39.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:31:39.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:31:39.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:31:39.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:31:39.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:31:40.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=16 2021-07-26T06:31:40.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:31:40.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:31:40.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:31:40.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 16 = 16 2021-07-26T06:31:40.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:31:40.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:31:40.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2021-07-26T06:31:40.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-07-26T06:31:40.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:31:40.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-07-26T06:31:40.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:40.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:40.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:40.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:40.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:40.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:40.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:31:40.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:31:40.597 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:31:40.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:31:40.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:31:40.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:31:40.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:40.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:31:40.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:31:40.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:40.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:31:40.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:31:40.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-07-26T06:31:40.639 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:31:40.639 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:40.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:31:40.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137330 2021-07-26T06:31:40.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:31:40.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:31:40.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:31:40.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-dup 0 2021-07-26T06:31:40.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-07-26T06:31:40.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:31:40.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-07-26T06:31:40.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:40.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:40.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:40.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:40.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:40.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:40.646 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:31:40.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:31:40.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:31:40.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:31:40.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:31:40.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:31:40.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:40.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:31:40.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:31:40.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:40.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:31:40.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:31:40.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-07-26T06:31:40.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:31:40.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:40.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.137330 2021-07-26T06:31:40.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.137330 2021-07-26T06:31:40.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:31:40.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:31:40.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:31:40.661 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:31:40.661 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:31:40.675 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2021-07-26T06:31:40.675 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2021-07-26T06:31:40.736 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:31:40.750 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:31:40.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:31:40.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:31:40.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:31:40.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:31:40.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:31:40.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:31:40.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:31:40.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:31:40.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-fast-mark-down 2021-07-26T06:31:40.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2021-07-26T06:31:40.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2021-07-26T06:31:40.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2021-07-26T06:31:40.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2021-07-26T06:31:40.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2021-07-26T06:31:40.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2021-07-26T06:31:40.755 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2021-07-26T06:31:40.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none ' 2021-07-26T06:31:40.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none ' 2021-07-26T06:31:40.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2021-07-26T06:31:40.756 INFO:tasks.workunit.client.0.smithi094.stdout:Ensuring old behavior is there... 2021-07-26T06:31:40.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2021-07-26T06:31:40.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2021-07-26T06:31:40.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-07-26T06:31:40.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-07-26T06:31:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-07-26T06:31:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-07-26T06:31:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:31:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-07-26T06:31:40.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:40.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:40.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:40.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:40.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:40.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:40.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:31:40.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:31:40.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:31:40.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:31:40.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:31:40.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:31:40.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:40.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:31:40.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:31:40.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:40.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:31:40.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:31:40.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-07-26T06:31:40.770 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:31:40.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:40.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:40.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.140654 2021-07-26T06:31:40.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:31:40.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:31:40.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-07-26T06:31:40.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:31:40.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:40.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:40.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.140654 2021-07-26T06:31:40.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:31:40.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:31:40.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:31:40.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:31:40.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-07-26T06:31:40.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-07-26T06:31:40.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-07-26T06:31:40.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:31:40.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:31:40.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:31:40.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-07-26T06:31:40.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-07-26T06:31:40.806 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:40.790+0000 7f70b6b01580 -1 failed for service _ceph-mon._tcp 2021-07-26T06:31:40.807 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:40.790+0000 7f70b6b01580 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-07-26T06:31:40.807 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:31:40.807 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:40.790+0000 7f70b6b01580 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2021-07-26T06:31:41.196 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:31:41.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:41.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:41.198 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:41.198 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:41.198 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:41.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:31:41.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-07-26T06:31:41.504 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:41.486+0000 7ff8206ae580 -1 failed for service _ceph-mon._tcp 2021-07-26T06:31:41.505 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:41.486+0000 7ff8206ae580 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-07-26T06:31:41.505 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:31:41.505 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:41.486+0000 7ff8206ae580 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2021-07-26T06:31:41.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2021-07-26T06:31:41.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2021-07-26T06:31:41.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-07-26T06:31:41.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-07-26T06:31:41.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-07-26T06:31:41.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2021-07-26T06:31:41.514 INFO:tasks.workunit.client.0.smithi094.stdout:Testing async msgr... 2021-07-26T06:31:41.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2021-07-26T06:31:41.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-07-26T06:31:41.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-07-26T06:31:41.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-07-26T06:31:41.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-07-26T06:31:41.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:31:41.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-07-26T06:31:41.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:31:41.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:31:41.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:31:41.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:31:41.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:31:41.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:31:41.519 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:31:41.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:31:41.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:31:41.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:31:41.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:31:41.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:31:41.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:41.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:31:41.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:31:41.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:31:41.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:31:41.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:31:41.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-07-26T06:31:41.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:31:41.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:41.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:41.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.140654 2021-07-26T06:31:41.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:31:41.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:31:41.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-07-26T06:31:41.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:31:41.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:41.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:41.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.140654 2021-07-26T06:31:41.538 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:31:41.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:31:41.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:31:41.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-07-26T06:31:41.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-07-26T06:31:41.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-07-26T06:31:41.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:31:41.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:31:41.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:31:41.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-07-26T06:31:41.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-07-26T06:31:41.976 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:31:41.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:41.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:41.978 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:41.978 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:41.978 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:41.979 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:31:41.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2021-07-26T06:31:42.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:31:42.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:31:42.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:31:42.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:31:42.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:31:42.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:31:42.302 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:31:42.302 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:31:42.302 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:31:42.303 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:31:42.303 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:42.303 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:42.304 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.140654/ceph-mon.a.asok 2021-07-26T06:31:42.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:31:42.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.140654/ceph-mon.a.asok config get fsid 2021-07-26T06:31:42.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:31:42.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:31:42.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:31:42.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:31:42.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:31:42.399 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:31:42.399 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:31:42.399 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:31:42.399 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:31:42.400 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:42.400 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:42.400 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.140654/ceph-mon.a.asok 2021-07-26T06:31:42.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:31:42.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.140654/ceph-mon.a.asok config get mon_host 2021-07-26T06:31:42.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2021-07-26T06:31:42.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2021-07-26T06:31:42.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:31:42.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:31:42.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:31:42.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2021-07-26T06:31:42.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:31:43.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:31:43.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:43.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:43.424 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:43.424 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:43.424 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:43.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:31:43.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:31:43.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:31:43.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-07-26T06:31:43.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2021-07-26T06:31:43.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-07-26T06:31:43.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:31:43.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:31:43.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:31:43.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2021-07-26T06:31:43.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-07-26T06:31:43.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:31:43.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:31:43.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:31:43.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2021-07-26T06:31:43.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2021-07-26T06:31:43.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:31:43.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:31:43.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:31:43.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-07-26T06:31:43.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:31:43.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:43.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:43.494 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:43.494 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:43.495 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:43.495 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:31:43.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:31:43.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:31:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:31:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:31:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-07-26T06:31:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-07-26T06:31:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:31:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:31:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:31:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:31:43.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2021-07-26T06:31:43.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:31:43.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74b08126-6555-46a9-a2e2-9cdf4842b3ae 2021-07-26T06:31:43.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 74b08126-6555-46a9-a2e2-9cdf4842b3ae' 2021-07-26T06:31:43.500 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 74b08126-6555-46a9-a2e2-9cdf4842b3ae 2021-07-26T06:31:43.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:31:43.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPVv5gtBu/HRAAtvp6rHXxqmMCMG7W39G7sQ== 2021-07-26T06:31:43.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPVv5gtBu/HRAAtvp6rHXxqmMCMG7W39G7sQ=="}' 2021-07-26T06:31:43.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74b08126-6555-46a9-a2e2-9cdf4842b3ae -i td/osd-fast-mark-down/0/new.json 2021-07-26T06:31:43.877 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:31:43.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2021-07-26T06:31:43.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDPVv5gtBu/HRAAtvp6rHXxqmMCMG7W39G7sQ== --osd-uuid 74b08126-6555-46a9-a2e2-9cdf4842b3ae 2021-07-26T06:31:43.915 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:43.898+0000 7fa1e102df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:43.917 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:43.898+0000 7fa1e102df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:43.919 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:43.902+0000 7fa1e102df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:43.920 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:43.902+0000 7fa1e102df40 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-07-26T06:31:43.920 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:43.902+0000 7fa1e102df40 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-07-26T06:31:43.920 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:43.902+0000 7fa1e102df40 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-07-26T06:31:44.007 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:43.990+0000 7fa1e102df40 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2021-07-26T06:31:49.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2021-07-26T06:31:49.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:31:49.937 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:31:49.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:31:49.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:31:50.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:31:50.444 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:31:50.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:50.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:31:50.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:31:50.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:31:50.468 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:50.450+0000 7f1e1840ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:50.471 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:50.454+0000 7f1e1840ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:50.482 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:50.454+0000 7f1e1840ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:50.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:31:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:31:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:31:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:31:50.925 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:31:50.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:31:50.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:50.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:31:50.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:50.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:51.036 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:51.018+0000 7f1e1840ef40 -1 Falling back to public interface 2021-07-26T06:31:51.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:52.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:52.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:52.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:31:52.315 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:31:52.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:52.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:52.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:53.437 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:53.418+0000 7f1e1840ef40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:31:53.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:53.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:53.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:31:53.723 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:31:53.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:53.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:54.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:31:55.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:31:55.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:31:55.084 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:31:55.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:31:55.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:31:55.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:31:55.497 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/141824,v1:127.0.0.1:6803/141824] [v2:127.0.0.1:6804/141824,v1:127.0.0.1:6805/141824] exists,up 74b08126-6555-46a9-a2e2-9cdf4842b3ae 2021-07-26T06:31:55.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:31:55.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:31:55.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:31:55.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2021-07-26T06:31:55.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=141824 2021-07-26T06:31:55.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-07-26T06:31:55.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2021-07-26T06:31:55.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-07-26T06:31:55.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:31:55.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:31:55.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:31:55.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2021-07-26T06:31:55.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-07-26T06:31:55.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:31:55.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:31:55.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:31:55.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2021-07-26T06:31:55.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2021-07-26T06:31:55.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:31:55.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:31:55.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:31:55.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-07-26T06:31:55.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:31:55.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:31:55.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:31:55.506 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:31:55.507 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:31:55.507 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:31:55.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:31:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:31:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:31:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:31:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:31:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-07-26T06:31:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-07-26T06:31:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:31:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:31:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:31:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:31:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:31:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2021-07-26T06:31:55.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:31:55.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=85c70aff-52f2-4d91-9aaf-2362bca6b8cd 2021-07-26T06:31:55.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 85c70aff-52f2-4d91-9aaf-2362bca6b8cd' 2021-07-26T06:31:55.512 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 85c70aff-52f2-4d91-9aaf-2362bca6b8cd 2021-07-26T06:31:55.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:31:55.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbVv5gq/t3HhAAxgznOnyZ+SriMHCcD/bpLA== 2021-07-26T06:31:55.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbVv5gq/t3HhAAxgznOnyZ+SriMHCcD/bpLA=="}' 2021-07-26T06:31:55.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 85c70aff-52f2-4d91-9aaf-2362bca6b8cd -i td/osd-fast-mark-down/1/new.json 2021-07-26T06:31:56.013 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:31:56.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2021-07-26T06:31:56.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDbVv5gq/t3HhAAxgznOnyZ+SriMHCcD/bpLA== --osd-uuid 85c70aff-52f2-4d91-9aaf-2362bca6b8cd 2021-07-26T06:31:56.053 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:56.034+0000 7f7dcae93f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:56.090 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:56.070+0000 7f7dcae93f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:56.091 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:56.074+0000 7f7dcae93f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:31:56.093 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:56.074+0000 7f7dcae93f40 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-07-26T06:31:56.093 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:56.074+0000 7f7dcae93f40 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-07-26T06:31:56.093 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:56.074+0000 7f7dcae93f40 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-07-26T06:31:56.135 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:31:56.118+0000 7f7dcae93f40 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2021-07-26T06:32:02.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2021-07-26T06:32:02.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:32:02.701 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:32:02.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:32:02.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:32:03.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:32:03.187 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:32:03.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:03.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:32:03.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:32:03.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:32:03.216 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:03.198+0000 7fb7103faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:03.218 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:03.202+0000 7fb7103faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:03.219 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:03.202+0000 7fb7103faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:03.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:32:03.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:03.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:32:03.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:03.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:03.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:03.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:03.605 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:32:03.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:03.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:03.759 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:03.742+0000 7fb7103faf40 -1 Falling back to public interface 2021-07-26T06:32:03.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:04.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:04.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:04.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:04.967 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:32:04.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:04.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:05.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:06.320 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:06.302+0000 7fb7103faf40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:32:06.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:06.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:06.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:06.472 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:32:06.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:06.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:07.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:08.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:08.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:08.015 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:32:08.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:08.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:08.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:08.380 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/143231,v1:127.0.0.1:6811/143231] [v2:127.0.0.1:6812/143231,v1:127.0.0.1:6813/143231] exists,up 85c70aff-52f2-4d91-9aaf-2362bca6b8cd 2021-07-26T06:32:08.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:08.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:08.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:08.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2021-07-26T06:32:08.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=143231 2021-07-26T06:32:08.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-07-26T06:32:08.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2021-07-26T06:32:08.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2021-07-26T06:32:08.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:32:08.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:32:08.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:32:08.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2021-07-26T06:32:08.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-07-26T06:32:08.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:08.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:08.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:08.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2021-07-26T06:32:08.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2021-07-26T06:32:08.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:08.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:08.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:32:08.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-07-26T06:32:08.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:32:08.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:08.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:08.392 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:32:08.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:32:08.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' 2021-07-26T06:32:08.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:08.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:32:08.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:32:08.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-07-26T06:32:08.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-07-26T06:32:08.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:32:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:32:08.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2021-07-26T06:32:08.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:32:08.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2f070437-33fc-46a6-8a55-1a755052aba8 2021-07-26T06:32:08.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2f070437-33fc-46a6-8a55-1a755052aba8' 2021-07-26T06:32:08.396 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 2f070437-33fc-46a6-8a55-1a755052aba8 2021-07-26T06:32:08.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:32:08.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoVv5gWuGEFxAA+DlK2FFQ+UnJaDZAnlcU5g== 2021-07-26T06:32:08.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoVv5gWuGEFxAA+DlK2FFQ+UnJaDZAnlcU5g=="}' 2021-07-26T06:32:08.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2f070437-33fc-46a6-8a55-1a755052aba8 -i td/osd-fast-mark-down/2/new.json 2021-07-26T06:32:08.982 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:32:08.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2021-07-26T06:32:08.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDoVv5gWuGEFxAA+DlK2FFQ+UnJaDZAnlcU5g== --osd-uuid 2f070437-33fc-46a6-8a55-1a755052aba8 2021-07-26T06:32:09.032 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:09.014+0000 7f9fcec56f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:09.055 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:09.038+0000 7f9fcec56f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:09.057 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:09.038+0000 7f9fcec56f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:09.058 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:09.038+0000 7f9fcec56f40 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-07-26T06:32:09.058 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:09.038+0000 7f9fcec56f40 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-07-26T06:32:09.059 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:09.038+0000 7f9fcec56f40 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-07-26T06:32:09.205 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:09.186+0000 7f9fcec56f40 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2021-07-26T06:32:15.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2021-07-26T06:32:15.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:32:15.945 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:32:15.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:32:15.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:32:16.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:32:16.448 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:32:16.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4d759caf-0ba9-4502-adbc-2ff5644f6547 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.140654/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:16.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:32:16.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:32:16.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:32:16.471 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:16.454+0000 7ff91abaff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:16.473 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:16.454+0000 7ff91abaff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:16.474 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:16.458+0000 7ff91abaff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:16.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:32:16.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:16.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:32:16.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:16.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:16.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:16.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:16.803 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:32:16.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:16.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:32:17.020 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:17.002+0000 7ff91abaff40 -1 Falling back to public interface 2021-07-26T06:32:17.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:18.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:18.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:18.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:18.142 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:32:18.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:18.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:32:18.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:19.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:19.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:19.500 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:32:19.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:19.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:19.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:32:19.751 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:19.734+0000 7ff91abaff40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:32:19.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:20.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:20.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:20.922 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:32:20.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:20.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:20.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:32:21.284 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/144638,v1:127.0.0.1:6819/144638] [v2:127.0.0.1:6820/144638,v1:127.0.0.1:6821/144638] exists,up 2f070437-33fc-46a6-8a55-1a755052aba8 2021-07-26T06:32:21.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:21.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:21.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:21.286 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2021-07-26T06:32:21.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=144638 2021-07-26T06:32:21.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2021-07-26T06:32:21.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:32:21.694 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' does not exist 2021-07-26T06:32:21.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:32:21.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:32:22.203 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' already exists 2021-07-26T06:32:22.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:32:23.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:32:25.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2021-07-26T06:32:25.310 INFO:tasks.workunit.client.0.smithi094.stdout:hints = 1 2021-07-26T06:32:25.311 INFO:tasks.workunit.client.0.smithi094.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2021-07-26T06:32:25.311 INFO:tasks.workunit.client.0.smithi094.stdout:Object prefix: benchmark_data_smithi094_145269 2021-07-26T06:32:25.311 INFO:tasks.workunit.client.0.smithi094.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-07-26T06:32:25.311 INFO:tasks.workunit.client.0.smithi094.stdout: 0 0 0 0 0 0 - 0 2021-07-26T06:32:26.310 INFO:tasks.workunit.client.0.smithi094.stdout: 1 16 65 49 0.191381 0.191406 0.268035 0.275082 2021-07-26T06:32:27.311 INFO:tasks.workunit.client.0.smithi094.stdout: 2 16 128 112 0.218716 0.246094 0.215835 0.27406 2021-07-26T06:32:28.311 INFO:tasks.workunit.client.0.smithi094.stdout:Total time run: 2.2113 2021-07-26T06:32:28.311 INFO:tasks.workunit.client.0.smithi094.stdout:Total writes made: 128 2021-07-26T06:32:28.312 INFO:tasks.workunit.client.0.smithi094.stdout:Write size: 4096 2021-07-26T06:32:28.312 INFO:tasks.workunit.client.0.smithi094.stdout:Object size: 4096 2021-07-26T06:32:28.312 INFO:tasks.workunit.client.0.smithi094.stdout:Bandwidth (MB/sec): 0.226111 2021-07-26T06:32:28.312 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Bandwidth: 0.0386699 2021-07-26T06:32:28.313 INFO:tasks.workunit.client.0.smithi094.stdout:Max bandwidth (MB/sec): 0.246094 2021-07-26T06:32:28.313 INFO:tasks.workunit.client.0.smithi094.stdout:Min bandwidth (MB/sec): 0.191406 2021-07-26T06:32:28.313 INFO:tasks.workunit.client.0.smithi094.stdout:Average IOPS: 57 2021-07-26T06:32:28.313 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev IOPS: 9.89949 2021-07-26T06:32:28.314 INFO:tasks.workunit.client.0.smithi094.stdout:Max IOPS: 63 2021-07-26T06:32:28.314 INFO:tasks.workunit.client.0.smithi094.stdout:Min IOPS: 49 2021-07-26T06:32:28.314 INFO:tasks.workunit.client.0.smithi094.stdout:Average Latency(s): 0.275309 2021-07-26T06:32:28.314 INFO:tasks.workunit.client.0.smithi094.stdout:Stddev Latency(s): 0.0327201 2021-07-26T06:32:28.314 INFO:tasks.workunit.client.0.smithi094.stdout:Max latency(s): 0.319819 2021-07-26T06:32:28.315 INFO:tasks.workunit.client.0.smithi094.stdout:Min latency(s): 0.117463 2021-07-26T06:32:28.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2021-07-26T06:32:29.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2021-07-26T06:32:29.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2021-07-26T06:32:29.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-07-26T06:32:29.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2021-07-26T06:32:29.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=143231 2021-07-26T06:32:29.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 143231 -eq 0 ']' 2021-07-26T06:32:29.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=143231 2021-07-26T06:32:29.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 143231 2021-07-26T06:32:29.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-07-26T06:32:29.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-07-26T06:32:29.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-07-26T06:32:29.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-07-26T06:32:30.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-07-26T06:32:30.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-07-26T06:32:30.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-07-26T06:32:30.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-07-26T06:32:30.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-07-26T06:32:30.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2021-07-26T06:32:30.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2021-07-26T06:32:30.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2021-07-26T06:32:30.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-07-26T06:32:30.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2021-07-26T06:32:30.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-07-26T06:32:30.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 143231 -eq 143231 ']' 2021-07-26T06:32:30.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=141824 2021-07-26T06:32:30.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 141824 -eq 143231 ']' 2021-07-26T06:32:30.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=141824 2021-07-26T06:32:30.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 141824 2021-07-26T06:32:30.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-07-26T06:32:30.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-07-26T06:32:30.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-07-26T06:32:30.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-07-26T06:32:31.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-07-26T06:32:31.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-07-26T06:32:31.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-07-26T06:32:31.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-07-26T06:32:31.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-07-26T06:32:32.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2021-07-26T06:32:32.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2021-07-26T06:32:32.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2021-07-26T06:32:32.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-07-26T06:32:32.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2021-07-26T06:32:32.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2021-07-26T06:32:32.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2021-07-26T06:32:32.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-07-26T06:32:32.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:32:32.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-07-26T06:32:32.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:32:32.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:32:32.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:32:32.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:32:32.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:32:32.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:32:32.431 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:32:32.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:32:32.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:32:32.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:32:32.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:32:32.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:32:32.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:32:32.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:32:32.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:32:32.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:32:32.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:32:32.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:32:32.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-07-26T06:32:32.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:32:32.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:32.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:32:32.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.140654 2021-07-26T06:32:32.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:32:32.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:32:32.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2021-07-26T06:32:32.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:32:32.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-fast-mark-down 0 2021-07-26T06:32:32.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-07-26T06:32:32.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:32:32.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-07-26T06:32:32.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:32:32.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:32:32.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:32:32.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:32:32.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:32:32.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:32:32.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:32:32.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:32:32.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:32:32.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:32:32.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:32:32.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:32:32.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:32:32.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:32:32.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:32:32.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:32:32.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:32:32.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:32:32.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-07-26T06:32:32.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:32:32.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:32.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.140654 2021-07-26T06:32:32.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.140654 2021-07-26T06:32:32.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:32:32.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:32:32.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:32:32.480 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:32:32.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:32:32.493 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2021-07-26T06:32:32.494 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2021-07-26T06:32:32.509 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:32:32.522 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:32:32.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:32:32.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:32:32.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:32:32.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:32:32.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:32:32.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:32:32.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:32:32.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:32:32.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-force-create-pg 2021-07-26T06:32:32.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2021-07-26T06:32:32.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2021-07-26T06:32:32.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2021-07-26T06:32:32.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2021-07-26T06:32:32.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2021-07-26T06:32:32.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2021-07-26T06:32:32.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none ' 2021-07-26T06:32:32.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2021-07-26T06:32:32.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2021-07-26T06:32:32.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:32:32.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2021-07-26T06:32:32.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2021-07-26T06:32:32.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2021-07-26T06:32:32.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2021-07-26T06:32:32.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2021-07-26T06:32:32.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-07-26T06:32:32.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:32:32.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-07-26T06:32:32.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:32:32.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:32:32.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:32:32.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:32:32.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:32:32.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:32:32.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:32:32.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:32:32.535 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:32:32.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:32:32.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:32:32.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:32:32.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:32:32.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:32:32.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:32:32.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:32:32.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:32:32.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:32:32.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-07-26T06:32:32.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:32:32.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:32.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:32.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145402 2021-07-26T06:32:32.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:32:32.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:32:32.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2021-07-26T06:32:32.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:32:32.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:32.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:32.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.145402 2021-07-26T06:32:32.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:32:32.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:32:32.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:32:32.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:32:32.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2021-07-26T06:32:32.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2021-07-26T06:32:32.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2021-07-26T06:32:32.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T06:32:32.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2021-07-26T06:32:32.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:32:32.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:32:32.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:32:32.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2021-07-26T06:32:32.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T06:32:32.966 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:32:32.966 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:32.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:32.968 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:32.968 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:32.968 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:32.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:32.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-07-26T06:32:33.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:32:33.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:32:33.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:32:33.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:32:33.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:32:33.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:32:33.326 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:32:33.327 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:32:33.327 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:32:33.327 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:32:33.327 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:33.327 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:33.328 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145402/ceph-mon.a.asok 2021-07-26T06:32:33.328 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:32:33.329 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.145402/ceph-mon.a.asok config get fsid 2021-07-26T06:32:33.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:32:33.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:32:33.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:32:33.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:32:33.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:32:33.416 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:32:33.416 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:32:33.416 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:32:33.417 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:32:33.417 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:33.418 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:33.418 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.145402/ceph-mon.a.asok 2021-07-26T06:32:33.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:32:33.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.145402/ceph-mon.a.asok config get mon_host 2021-07-26T06:32:33.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2021-07-26T06:32:33.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2021-07-26T06:32:33.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:32:33.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:32:33.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:32:33.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2021-07-26T06:32:33.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:32:34.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:32:34.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:34.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:34.394 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:34.394 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:34.394 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:34.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:34.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:32:34.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:32:34.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2021-07-26T06:32:34.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-07-26T06:32:34.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:32:34.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:32:34.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:32:34.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2021-07-26T06:32:34.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-07-26T06:32:34.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:34.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:34.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:34.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-07-26T06:32:34.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-07-26T06:32:34.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:34.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:34.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:32:34.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-07-26T06:32:34.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:32:34.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:34.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:34.424 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:34.425 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:34.425 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:34.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:34.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:34.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:34.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:32:34.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:32:34.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-07-26T06:32:34.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-07-26T06:32:34.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:34.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:34.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:34.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:32:34.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:32:34.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2021-07-26T06:32:34.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:32:34.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7a809a5c-83da-43f8-8611-bf19c331a460 2021-07-26T06:32:34.429 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 7a809a5c-83da-43f8-8611-bf19c331a460 2021-07-26T06:32:34.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7a809a5c-83da-43f8-8611-bf19c331a460' 2021-07-26T06:32:34.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:32:34.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACV/5gQcE2GRAAM/WhJ9GGhgEzJp9wmL3AuQ== 2021-07-26T06:32:34.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACV/5gQcE2GRAAM/WhJ9GGhgEzJp9wmL3AuQ=="}' 2021-07-26T06:32:34.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7a809a5c-83da-43f8-8611-bf19c331a460 -i td/osd-force-create-pg/0/new.json 2021-07-26T06:32:34.800 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:32:34.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2021-07-26T06:32:34.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQACV/5gQcE2GRAAM/WhJ9GGhgEzJp9wmL3AuQ== --osd-uuid 7a809a5c-83da-43f8-8611-bf19c331a460 2021-07-26T06:32:34.844 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:34.826+0000 7f84a9a0af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:34.846 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:34.830+0000 7f84a9a0af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:34.849 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:34.830+0000 7f84a9a0af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:34.849 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:34.830+0000 7f84a9a0af40 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-07-26T06:32:34.849 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:34.830+0000 7f84a9a0af40 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-07-26T06:32:34.850 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:34.830+0000 7f84a9a0af40 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-07-26T06:32:34.926 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:34.906+0000 7f84a9a0af40 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2021-07-26T06:32:40.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2021-07-26T06:32:40.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:32:40.829 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:32:40.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:32:40.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:32:41.324 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:32:41.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:32:41.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:41.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:32:41.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:32:41.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:32:41.353 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:41.334+0000 7fbfbca8af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:41.355 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:41.338+0000 7fbfbca8af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:41.357 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:41.338+0000 7fbfbca8af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:41.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:32:41.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:41.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:32:41.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:41.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:41.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:41.775 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:32:41.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:41.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:41.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:41.911 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:41.894+0000 7fbfbca8af40 -1 Falling back to public interface 2021-07-26T06:32:42.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:43.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:43.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:43.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:43.132 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:32:43.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:43.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:43.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:44.294 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:44.274+0000 7fbfbca8af40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:32:44.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:44.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:44.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:44.492 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:32:44.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:44.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:44.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:45.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:45.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:45.925 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:32:45.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:45.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:45.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:32:46.276 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/146506,v1:127.0.0.1:6803/146506] [v2:127.0.0.1:6804/146506,v1:127.0.0.1:6805/146506] exists,up 7a809a5c-83da-43f8-8611-bf19c331a460 2021-07-26T06:32:46.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:46.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:46.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:46.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2021-07-26T06:32:46.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-07-26T06:32:46.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:32:46.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:32:46.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:32:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2021-07-26T06:32:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-07-26T06:32:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:46.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-07-26T06:32:46.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-07-26T06:32:46.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:46.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:46.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:32:46.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-07-26T06:32:46.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:32:46.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:46.286 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:46.286 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:46.286 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:46.286 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:46.287 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:46.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:46.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:46.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:32:46.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:32:46.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-07-26T06:32:46.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-07-26T06:32:46.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:46.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:46.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:46.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:32:46.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:32:46.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2021-07-26T06:32:46.290 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:32:46.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c7721f21-b5ab-422f-8f95-f6521404378c 2021-07-26T06:32:46.291 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 c7721f21-b5ab-422f-8f95-f6521404378c 2021-07-26T06:32:46.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c7721f21-b5ab-422f-8f95-f6521404378c' 2021-07-26T06:32:46.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:32:46.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOV/5gjLVEERAAXZ57Gfvn0d1gsMbcgxeiXg== 2021-07-26T06:32:46.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOV/5gjLVEERAAXZ57Gfvn0d1gsMbcgxeiXg=="}' 2021-07-26T06:32:46.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c7721f21-b5ab-422f-8f95-f6521404378c -i td/osd-force-create-pg/1/new.json 2021-07-26T06:32:47.003 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:32:47.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2021-07-26T06:32:47.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAOV/5gjLVEERAAXZ57Gfvn0d1gsMbcgxeiXg== --osd-uuid c7721f21-b5ab-422f-8f95-f6521404378c 2021-07-26T06:32:47.050 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:47.030+0000 7f8db0188f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:47.060 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:47.042+0000 7f8db0188f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:47.062 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:47.046+0000 7f8db0188f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:47.063 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:47.046+0000 7f8db0188f40 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-07-26T06:32:47.063 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:47.046+0000 7f8db0188f40 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-07-26T06:32:47.063 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:47.046+0000 7f8db0188f40 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-07-26T06:32:47.152 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:47.134+0000 7f8db0188f40 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2021-07-26T06:32:53.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2021-07-26T06:32:53.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:32:53.749 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:32:53.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:32:53.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:32:54.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:32:54.244 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:32:54.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:54.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:32:54.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:32:54.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:32:54.269 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:54.249+0000 7fc7e4447f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:54.272 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:54.253+0000 7fc7e4447f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:54.273 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:54.253+0000 7fc7e4447f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:54.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:32:54.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:32:54.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:32:54.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:32:54.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:32:54.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:54.613 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:32:54.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:32:54.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:54.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:54.808 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:54.789+0000 7fc7e4447f40 -1 Falling back to public interface 2021-07-26T06:32:54.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:55.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:55.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:55.987 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:32:55.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:32:55.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:55.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:56.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:57.257 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:57.237+0000 7fc7e4447f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:32:57.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:57.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:57.342 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:32:57.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:32:57.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:57.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:57.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:32:58.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:32:58.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:32:58.807 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:32:58.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:32:58.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:32:58.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:32:59.165 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/147912,v1:127.0.0.1:6811/147912] [v2:127.0.0.1:6812/147912,v1:127.0.0.1:6813/147912] exists,up c7721f21-b5ab-422f-8f95-f6521404378c 2021-07-26T06:32:59.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:32:59.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:32:59.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:32:59.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2021-07-26T06:32:59.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2021-07-26T06:32:59.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:32:59.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:32:59.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:32:59.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2021-07-26T06:32:59.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-07-26T06:32:59.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:32:59.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:32:59.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:32:59.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-07-26T06:32:59.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-07-26T06:32:59.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:32:59.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:32:59.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:32:59.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-07-26T06:32:59.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:32:59.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:32:59.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:32:59.175 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:32:59.175 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:32:59.175 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:32:59.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:59.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:32:59.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:32:59.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:32:59.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:32:59.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-07-26T06:32:59.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-07-26T06:32:59.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:32:59.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:32:59.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:32:59.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:32:59.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:32:59.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2021-07-26T06:32:59.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:32:59.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a2b1416-b089-45a1-8e9d-71d12ba0c250 2021-07-26T06:32:59.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2a2b1416-b089-45a1-8e9d-71d12ba0c250' 2021-07-26T06:32:59.180 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 2a2b1416-b089-45a1-8e9d-71d12ba0c250 2021-07-26T06:32:59.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:32:59.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbV/5gkUV9ChAA1nhMyERGYHPr+Azclk34qw== 2021-07-26T06:32:59.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbV/5gkUV9ChAA1nhMyERGYHPr+Azclk34qw=="}' 2021-07-26T06:32:59.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a2b1416-b089-45a1-8e9d-71d12ba0c250 -i td/osd-force-create-pg/2/new.json 2021-07-26T06:32:59.790 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:32:59.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2021-07-26T06:32:59.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAbV/5gkUV9ChAA1nhMyERGYHPr+Azclk34qw== --osd-uuid 2a2b1416-b089-45a1-8e9d-71d12ba0c250 2021-07-26T06:32:59.836 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:59.817+0000 7f816a94df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:59.838 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:59.821+0000 7f816a94df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:59.839 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:59.821+0000 7f816a94df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:32:59.840 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:59.821+0000 7f816a94df40 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-07-26T06:32:59.840 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:59.821+0000 7f816a94df40 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-07-26T06:32:59.841 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:59.821+0000 7f816a94df40 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-07-26T06:32:59.925 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:32:59.909+0000 7f816a94df40 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2021-07-26T06:33:06.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2021-07-26T06:33:06.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:33:06.615 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:33:06.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:33:06.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:33:07.138 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:33:07.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:33:07.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:07.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:33:07.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:33:07.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:33:07.166 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:07.149+0000 7fd5bd792f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:07.168 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:07.149+0000 7fd5bd792f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:07.169 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:07.149+0000 7fd5bd792f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:07.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:33:07.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:07.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:33:07.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:07.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:07.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:07.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:07.494 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:33:07.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:07.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:07.704 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:07.685+0000 7fd5bd792f40 -1 Falling back to public interface 2021-07-26T06:33:07.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:08.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:08.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:08.856 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:33:08.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:08.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:08.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:09.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:10.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:10.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:10.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:10.321 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:33:10.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:10.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:10.340 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:10.321+0000 7fd5bd792f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:33:10.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:11.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:11.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:11.836 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:33:11.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:33:11.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:11.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:12.197 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/149318,v1:127.0.0.1:6819/149318] [v2:127.0.0.1:6820/149318,v1:127.0.0.1:6821/149318] exists,up 2a2b1416-b089-45a1-8e9d-71d12ba0c250 2021-07-26T06:33:12.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:12.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:12.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:12.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2021-07-26T06:33:12.843 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'foo' already exists 2021-07-26T06:33:12.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2021-07-26T06:33:12.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:33:12.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:33:12.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:33:12.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:33:12.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:33:12.857 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:33:12.857 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:33:12.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:33:12.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:33:12.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:33:13.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:33:13.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:33:13.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:33:13.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:33:13.063 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:33:13.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:33:13.763 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:33:13.763 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:33:13.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:33:13.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:13.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:33:14.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T06:33:14.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T06:33:14.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T06:33:14.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:14.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:33:14.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T06:33:14.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T06:33:14.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-07-26T06:33:14.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:14.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:33:14.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509443 2021-07-26T06:33:14.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509443 2021-07-26T06:33:14.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509443' 2021-07-26T06:33:14.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:14.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T06:33:14.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:14.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:33:14.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T06:33:14.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:14.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T06:33:14.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T06:33:14.449 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836488 2021-07-26T06:33:14.450 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:14.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T06:33:14.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:14.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T06:33:14.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:14.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:33:14.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T06:33:14.875 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:14.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T06:33:14.877 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672966 2021-07-26T06:33:14.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T06:33:14.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:33:15.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672966 2021-07-26T06:33:15.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:15.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509443 2021-07-26T06:33:15.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:15.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:33:15.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509443 2021-07-26T06:33:15.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:15.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509443 2021-07-26T06:33:15.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-07-26T06:33:15.341 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509443 2021-07-26T06:33:15.341 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:33:15.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509441 -lt 64424509443 2021-07-26T06:33:15.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:33:16.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:33:16.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:33:17.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509443 -lt 64424509443 2021-07-26T06:33:17.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:33:17.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:33:17.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:33:17.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 50 == 0 2021-07-26T06:33:17.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:33:17.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:33:17.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:33:17.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:33:17.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:33:17.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:33:17.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:33:17.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=50 2021-07-26T06:33:17.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:33:17.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:33:17.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:33:18.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 50 = 50 2021-07-26T06:33:18.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:33:18.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:33:18.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2021-07-26T06:33:18.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:18.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:18.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:18.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:18.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:18.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:18.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2021-07-26T06:33:18.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:18.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:18.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:18.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:18.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:18.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:18.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2021-07-26T06:33:18.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:18.669 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:18.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:18.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:18.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:18.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:18.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2021-07-26T06:33:20.120 INFO:tasks.workunit.client.0.smithi094.stderr:PG '1.0' not found 2021-07-26T06:33:20.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2021-07-26T06:33:22.002 INFO:tasks.workunit.client.0.smithi094.stdout: marking collection for removal 2021-07-26T06:33:22.003 INFO:tasks.workunit.client.0.smithi094.stdout:setting '_remove' omap key 2021-07-26T06:33:22.003 INFO:tasks.workunit.client.0.smithi094.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-07-26T06:33:22.142 INFO:tasks.workunit.client.0.smithi094.stdout:Remove successful 2021-07-26T06:33:22.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2021-07-26T06:33:24.019 INFO:tasks.workunit.client.0.smithi094.stderr:PG '1.0' not found 2021-07-26T06:33:24.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2021-07-26T06:33:24.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-force-create-pg 2021-07-26T06:33:24.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:33:24.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:33:24.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:33:24.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-force-create-pg/0 2021-07-26T06:33:24.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-07-26T06:33:24.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:24.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:24.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:24.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-07-26T06:33:24.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-07-26T06:33:24.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:24.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:24.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:33:24.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-07-26T06:33:24.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:33:24.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:24.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:24.563 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:24.563 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:24.563 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:33:24.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:33:24.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:33:24.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:24.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-07-26T06:33:24.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-07-26T06:33:24.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:24.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:24.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:24.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:33:24.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:33:24.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-force-create-pg/0 2021-07-26T06:33:24.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:33:24.569 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:33:24.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:24.570 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-force-create-pg/0/whoami 2021-07-26T06:33:24.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:33:24.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:33:24.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:33:24.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:33:24.591 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:24.573+0000 7fd55ccc1f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:24.613 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:24.593+0000 7fd55ccc1f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:24.615 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:24.597+0000 7fd55ccc1f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:24.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:33:24.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:24.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:33:24.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:24.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:24.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:24.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:24.935 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:33:24.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:24.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:25.164 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:25.145+0000 7fd55ccc1f40 -1 Falling back to public interface 2021-07-26T06:33:25.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:26.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:26.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:26.296 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:33:26.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:26.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:26.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:26.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:27.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:27.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:27.677 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:33:27.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:27.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:27.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:27.932 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:27.913+0000 7fd55ccc1f40 -1 osd.0 22 log_to_monitors {default=true} 2021-07-26T06:33:28.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:29.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:29.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:29.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:33:29.048 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:33:29.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:29.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:33:29.527 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/151494,v1:127.0.0.1:6803/151494] [v2:127.0.0.1:6804/151494,v1:127.0.0.1:6805/151494] exists,up 7a809a5c-83da-43f8-8611-bf19c331a460 2021-07-26T06:33:29.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:29.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:29.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:29.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2021-07-26T06:33:29.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-force-create-pg 2021-07-26T06:33:29.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:33:29.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T06:33:29.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:33:29.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-force-create-pg/1 2021-07-26T06:33:29.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-07-26T06:33:29.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:29.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:29.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:29.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-07-26T06:33:29.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-07-26T06:33:29.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:29.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:29.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:33:29.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-07-26T06:33:29.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:33:29.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:29.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:29.534 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:29.534 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:29.535 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:33:29.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:33:29.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:33:29.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:29.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-07-26T06:33:29.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-07-26T06:33:29.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:29.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:29.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:29.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:33:29.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:33:29.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-force-create-pg/1 2021-07-26T06:33:29.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T06:33:29.541 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:33:29.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-force-create-pg/1/whoami 2021-07-26T06:33:29.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:29.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T06:33:29.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:33:29.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:33:29.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:33:29.561 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:29.545+0000 7f1cf23edf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:29.579 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:29.561+0000 7f1cf23edf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:29.581 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:29.561+0000 7f1cf23edf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:29.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T06:33:29.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:29.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:33:29.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:29.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:29.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:29.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:29.910 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:33:29.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:29.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:30.132 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:30.113+0000 7f1cf23edf40 -1 Falling back to public interface 2021-07-26T06:33:30.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:31.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:31.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:31.264 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:33:31.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:31.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:31.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:31.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:32.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:32.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:32.623 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:33:32.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:32.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:32.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:32.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:33.207 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:33.189+0000 7f1cf23edf40 -1 osd.1 22 log_to_monitors {default=true} 2021-07-26T06:33:33.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:33.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:33.986 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:33:33.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:33:33.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:33.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:33:34.564 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,22) [v2:127.0.0.1:6810/152072,v1:127.0.0.1:6811/152072] [v2:127.0.0.1:6812/152072,v1:127.0.0.1:6813/152072] exists,up c7721f21-b5ab-422f-8f95-f6521404378c 2021-07-26T06:33:34.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:34.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:34.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:34.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2021-07-26T06:33:34.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-force-create-pg 2021-07-26T06:33:34.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:33:34.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=2 2021-07-26T06:33:34.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:33:34.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-force-create-pg/2 2021-07-26T06:33:34.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-07-26T06:33:34.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:34.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:34.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:34.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-07-26T06:33:34.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-07-26T06:33:34.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:34.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:34.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:33:34.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-07-26T06:33:34.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:33:34.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:34.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:34.573 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:34.573 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:34.574 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:33:34.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:33:34.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' 2021-07-26T06:33:34.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:34.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-07-26T06:33:34.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-07-26T06:33:34.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:34.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:34.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:34.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:33:34.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:33:34.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-force-create-pg/2 2021-07-26T06:33:34.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.2 2021-07-26T06:33:34.577 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:33:34.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 2 --fsid=d12b31d5-57b9-47e2-b5cb-0598f655b20f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.145402/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:34.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-force-create-pg/2/whoami 2021-07-26T06:33:34.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 2 = 2 ']' 2021-07-26T06:33:34.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:33:34.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:33:34.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:33:34.597 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:34.577+0000 7faff5f81f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:34.683 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:34.665+0000 7faff5f81f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:34.685 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:34.665+0000 7faff5f81f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:34.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 2 2021-07-26T06:33:34.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:33:34.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:33:34.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:33:34.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:33:34.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:34.965 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:33:34.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:33:34.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:34.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:35.003 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:34.985+0000 7faff5f81f40 -1 Falling back to public interface 2021-07-26T06:33:35.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:36.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:36.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:36.362 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:33:36.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:33:36.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:36.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:36.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:33:37.580 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:37.561+0000 7faff5f81f40 -1 osd.2 22 log_to_monitors {default=true} 2021-07-26T06:33:37.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:33:37.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:33:37.837 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:33:37.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:33:37.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:33:37.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:33:38.363 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 32 up_thru 17 down_at 27 last_clean_interval [15,22) [v2:127.0.0.1:6818/152668,v1:127.0.0.1:6819/152668] [v2:127.0.0.1:6820/152668,v1:127.0.0.1:6821/152668] exists,up 2a2b1416-b089-45a1-8e9d-71d12ba0c250 2021-07-26T06:33:38.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:33:38.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:33:38.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:33:38.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2021-07-26T06:33:48.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2021-07-26T06:33:48.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2021-07-26T06:33:48.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2021-07-26T06:33:48.721 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 stale+active+clean 34s 0'0 19:9 [1]p1 [1]p1 2021-07-26T06:33:12.465586+0000 2021-07-26T06:33:12.465586+0000 2021-07-26T06:33:48.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2021-07-26T06:33:49.682 INFO:tasks.workunit.client.0.smithi094.stderr:pg 1.0 already creating 2021-07-26T06:33:49.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2021-07-26T06:33:49.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:33:49.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:33:49.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:33:49.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:33:49.697 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:33:49.697 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:33:49.698 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:33:49.698 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:33:49.698 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:33:49.699 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:33:49.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:33:49.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:33:49.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:33:49.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:33:49.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:33:50.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:33:50.338 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:33:50.338 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T06:33:50.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:33:50.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:50.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:33:50.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=111669149703 2021-07-26T06:33:50.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 111669149703 2021-07-26T06:33:50.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149703' 2021-07-26T06:33:50.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:50.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:33:50.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=124554051590 2021-07-26T06:33:50.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 124554051590 2021-07-26T06:33:50.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149703 1-124554051590' 2021-07-26T06:33:50.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:33:50.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:33:50.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=137438953477 2021-07-26T06:33:50.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 137438953477 2021-07-26T06:33:50.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-111669149703 1-124554051590 2-137438953477' 2021-07-26T06:33:50.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:50.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-111669149703 2021-07-26T06:33:50.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:50.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:33:50.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-111669149703 2021-07-26T06:33:50.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:50.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=111669149703 2021-07-26T06:33:50.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 111669149703' 2021-07-26T06:33:50.756 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 111669149703 2021-07-26T06:33:50.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:51.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149701 -lt 111669149703 2021-07-26T06:33:51.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:33:52.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:33:52.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:52.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149701 -lt 111669149703 2021-07-26T06:33:52.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:33:53.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:33:53.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:33:53.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 111669149703 -lt 111669149703 2021-07-26T06:33:53.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:53.829 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-124554051590 2021-07-26T06:33:53.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:53.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:33:53.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-124554051590 2021-07-26T06:33:53.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:53.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=124554051590 2021-07-26T06:33:53.834 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 124554051590 2021-07-26T06:33:53.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 124554051590' 2021-07-26T06:33:53.835 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:33:54.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 124554051590 -lt 124554051590 2021-07-26T06:33:54.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:33:54.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-137438953477 2021-07-26T06:33:54.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:33:54.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:33:54.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-137438953477 2021-07-26T06:33:54.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:33:54.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=137438953477 2021-07-26T06:33:54.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 137438953477' 2021-07-26T06:33:54.196 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 137438953477 2021-07-26T06:33:54.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:33:54.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 137438953478 -lt 137438953477 2021-07-26T06:33:54.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:33:54.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:33:54.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:33:55.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 50 == 0 2021-07-26T06:33:55.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:33:55.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:33:55.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:33:55.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:33:55.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:33:55.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:33:55.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:33:55.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=50 2021-07-26T06:33:55.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:33:55.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:33:55.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:33:55.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 50 = 50 2021-07-26T06:33:55.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:33:55.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:33:55.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2021-07-26T06:33:55.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-07-26T06:33:55.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:33:55.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-07-26T06:33:55.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:55.905 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:55.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:55.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:55.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:56.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:56.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:33:56.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:33:56.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:33:56.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:33:56.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:33:56.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:33:56.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:56.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:33:56.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:33:56.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:56.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:33:56.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:33:56.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-07-26T06:33:56.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:33:56.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:56.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:33:56.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145402 2021-07-26T06:33:56.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:33:56.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:33:56.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:33:56.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-force-create-pg 0 2021-07-26T06:33:56.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-07-26T06:33:56.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:33:56.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-07-26T06:33:56.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:56.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:56.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:56.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:56.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:56.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:56.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:33:56.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:33:56.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:33:56.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:33:56.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:33:56.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:33:56.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:56.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:33:56.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:33:56.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:56.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:33:56.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:33:56.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-07-26T06:33:56.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:33:56.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:56.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.145402 2021-07-26T06:33:56.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.145402 2021-07-26T06:33:56.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:33:56.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:33:56.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:33:56.066 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:33:56.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:33:56.122 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2021-07-26T06:33:56.122 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2021-07-26T06:33:56.139 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:33:56.154 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:33:56.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:33:56.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:33:56.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:33:56.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:33:56.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:33:56.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:33:56.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:33:56.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:33:56.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-markdown 2021-07-26T06:33:56.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2021-07-26T06:33:56.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2021-07-26T06:33:56.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2021-07-26T06:33:56.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2021-07-26T06:33:56.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2021-07-26T06:33:56.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2021-07-26T06:33:56.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none ' 2021-07-26T06:33:56.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2021-07-26T06:33:56.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2021-07-26T06:33:56.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:33:56.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2021-07-26T06:33:56.161 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_markdown_boot_exceed_time 2021-07-26T06:33:56.161 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_markdown_exceed_maxdown_count 2021-07-26T06:33:56.161 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_osd_stop' 2021-07-26T06:33:56.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-07-26T06:33:56.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-07-26T06:33:56.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-07-26T06:33:56.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-07-26T06:33:56.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:33:56.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:33:56.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:33:56.163 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:33:56.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:33:56.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:33:56.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:33:56.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:33:56.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:33:56.166 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:33:56.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:33:56.166 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:33:56.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:33:56.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:33:56.169 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:33:56.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:56.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:33:56.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:33:56.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:33:56.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:33:56.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:33:56.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:33:56.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:33:56.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:56.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:33:56.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:33:56.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:33:56.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:33:56.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-07-26T06:33:56.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:33:56.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:56.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:33:56.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153711 2021-07-26T06:33:56.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:33:56.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:33:56.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:33:56.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:33:56.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-07-26T06:33:56.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2021-07-26T06:33:56.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2021-07-26T06:33:56.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2021-07-26T06:33:56.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-07-26T06:33:56.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:33:56.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:33:56.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:33:56.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-07-26T06:33:56.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-07-26T06:33:56.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:33:56.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:56.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:56.655 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:56.656 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:56.656 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:33:56.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:33:56.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:33:56.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:33:56.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:33:56.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:33:56.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:33:56.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:33:56.979 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:33:56.980 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:33:56.980 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:33:56.980 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:33:56.981 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:33:56.981 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:56.981 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:33:56.981 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:33:56.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:33:56.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get fsid 2021-07-26T06:33:57.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:33:57.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:33:57.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:33:57.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:33:57.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:33:57.068 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:33:57.068 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:33:57.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:33:57.069 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:33:57.070 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:57.070 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:33:57.070 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:33:57.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:33:57.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get mon_host 2021-07-26T06:33:57.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2021-07-26T06:33:57.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-07-26T06:33:57.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:33:57.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:33:57.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:33:57.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-07-26T06:33:57.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:33:58.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:33:58.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:58.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:58.046 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:58.046 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:58.046 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:33:58.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:33:58.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:33:58.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:33:58.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2021-07-26T06:33:58.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:33:58.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:33:58.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:33:58.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:33:58.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-07-26T06:33:58.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:33:58.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:33:58.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:33:58.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:33:58.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-07-26T06:33:58.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-07-26T06:33:58.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:33:58.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:33:58.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:33:58.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:33:58.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:33:58.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:33:58.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:33:58.082 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:33:58.082 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:33:58.083 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:33:58.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:33:58.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:33:58.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:33:58.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:33:58.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:33:58.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:33:58.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:33:58.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:33:58.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:33:58.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:33:58.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:33:58.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:33:58.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2021-07-26T06:33:58.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:33:58.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43138fa7-527b-439a-98ea-89eee22e9378 2021-07-26T06:33:58.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 43138fa7-527b-439a-98ea-89eee22e9378' 2021-07-26T06:33:58.087 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 43138fa7-527b-439a-98ea-89eee22e9378 2021-07-26T06:33:58.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:33:58.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWV/5gGyX8BBAAd25Fg1nivR7FvMe7ambOLg== 2021-07-26T06:33:58.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWV/5gGyX8BBAAd25Fg1nivR7FvMe7ambOLg=="}' 2021-07-26T06:33:58.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43138fa7-527b-439a-98ea-89eee22e9378 -i td/osd-markdown/0/new.json 2021-07-26T06:33:58.653 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:33:58.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2021-07-26T06:33:58.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWV/5gGyX8BBAAd25Fg1nivR7FvMe7ambOLg== --osd-uuid 43138fa7-527b-439a-98ea-89eee22e9378 2021-07-26T06:33:58.697 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:58.676+0000 7f0e240b7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:58.739 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:58.720+0000 7f0e240b7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:58.741 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:58.720+0000 7f0e240b7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:33:58.742 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:58.724+0000 7f0e240b7f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:33:58.743 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:58.724+0000 7f0e240b7f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:33:58.743 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:58.724+0000 7f0e240b7f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:33:58.788 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:33:58.768+0000 7f0e240b7f40 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-07-26T06:34:04.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-07-26T06:34:04.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:34:04.315 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:34:04.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:34:04.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:34:04.801 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:34:04.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:34:04.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:04.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:34:04.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:34:04.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:34:04.827 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:04.808+0000 7f51ac500f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:04.829 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:04.808+0000 7f51ac500f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:04.830 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:04.812+0000 7f51ac500f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:34:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:34:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:34:05.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:34:05.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:34:05.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:05.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:34:05.154 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:34:05.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:05.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:05.371 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:05.352+0000 7f51ac500f40 -1 Falling back to public interface 2021-07-26T06:34:05.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:06.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:06.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:06.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:34:06.514 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:34:06.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:06.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:06.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:07.855 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:07.836+0000 7f51ac500f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:34:07.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:07.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:07.876 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:34:07.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:34:07.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:07.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:08.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:09.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:09.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:09.339 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:34:09.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:34:09.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:09.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:34:09.711 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/154815,v1:127.0.0.1:6803/154815] [v2:127.0.0.1:6804/154815,v1:127.0.0.1:6805/154815] exists,up 43138fa7-527b-439a-98ea-89eee22e9378 2021-07-26T06:34:09.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:34:09.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:34:09.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:34:09.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2021-07-26T06:34:09.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:34:09.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:34:09.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:34:09.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:34:09.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-07-26T06:34:09.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:34:09.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:34:09.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:34:09.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:34:09.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-07-26T06:34:09.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-07-26T06:34:09.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:34:09.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:34:09.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:34:09.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:34:09.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:34:09.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:34:09.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:34:09.720 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:34:09.720 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:34:09.720 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:34:09.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:34:09.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:34:09.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:34:09.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:34:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:34:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:34:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:34:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:34:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:34:09.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:09.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:34:09.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:34:09.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2021-07-26T06:34:09.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:34:09.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dfd693ca-5144-4c9d-829c-60d8b1f096b5 2021-07-26T06:34:09.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dfd693ca-5144-4c9d-829c-60d8b1f096b5' 2021-07-26T06:34:09.725 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 dfd693ca-5144-4c9d-829c-60d8b1f096b5 2021-07-26T06:34:09.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:34:09.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhV/5gWP33KhAAPpqML5Df8lPIqXzb1gPSxw== 2021-07-26T06:34:09.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhV/5gWP33KhAAPpqML5Df8lPIqXzb1gPSxw=="}' 2021-07-26T06:34:09.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dfd693ca-5144-4c9d-829c-60d8b1f096b5 -i td/osd-markdown/1/new.json 2021-07-26T06:34:10.246 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:34:10.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2021-07-26T06:34:10.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBhV/5gWP33KhAAPpqML5Df8lPIqXzb1gPSxw== --osd-uuid dfd693ca-5144-4c9d-829c-60d8b1f096b5 2021-07-26T06:34:10.286 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:10.268+0000 7fe101dd9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:10.301 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:10.280+0000 7fe101dd9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:10.303 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:10.284+0000 7fe101dd9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:10.303 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:10.284+0000 7fe101dd9f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:34:10.304 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:10.284+0000 7fe101dd9f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:34:10.304 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:10.284+0000 7fe101dd9f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:34:10.396 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:10.376+0000 7fe101dd9f40 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-07-26T06:34:16.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-07-26T06:34:16.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:34:16.236 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:34:16.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:34:16.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:34:16.708 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:34:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:34:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:34:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:34:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:34:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:16.716+0000 7fb04b45ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:16.822 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:16.720+0000 7fb04b45ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:16.823 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:16.720+0000 7fb04b45ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:17.020 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:17.000+0000 7fb04b45ef40 -1 Falling back to public interface 2021-07-26T06:34:17.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:34:17.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:34:17.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:34:17.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:34:17.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:34:17.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:17.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:34:17.054 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:34:17.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:17.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:17.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:18.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:18.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:18.418 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:34:18.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:34:18.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:18.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:18.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:19.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:19.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:19.771 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:34:19.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:34:19.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:19.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:19.822 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:19.804+0000 7fb04b45ef40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:34:20.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:21.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:21.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:21.146 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:34:21.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:34:21.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:21.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:34:21.509 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/156221,v1:127.0.0.1:6811/156221] [v2:127.0.0.1:6812/156221,v1:127.0.0.1:6813/156221] exists,up dfd693ca-5144-4c9d-829c-60d8b1f096b5 2021-07-26T06:34:21.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:34:21.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:34:21.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:34:21.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2021-07-26T06:34:21.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:34:21.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:34:21.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:34:21.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:34:21.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-07-26T06:34:21.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:34:21.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:34:21.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:34:21.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:34:21.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-07-26T06:34:21.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-07-26T06:34:21.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:34:21.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:34:21.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:34:21.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:34:21.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:34:21.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:34:21.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:34:21.517 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:34:21.517 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:34:21.517 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:34:21.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:34:21.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:34:21.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:34:21.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:34:21.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:34:21.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:34:21.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:34:21.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:34:21.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:34:21.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:21.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:34:21.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:34:21.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2021-07-26T06:34:21.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:34:21.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=362a6d0c-1de4-4cbf-892c-802ffe122833 2021-07-26T06:34:21.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 362a6d0c-1de4-4cbf-892c-802ffe122833' 2021-07-26T06:34:21.523 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 362a6d0c-1de4-4cbf-892c-802ffe122833 2021-07-26T06:34:21.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:34:21.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtV/5gJnslHxAAZa+ojLHeaEepklUQZriy4A== 2021-07-26T06:34:21.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtV/5gJnslHxAAZa+ojLHeaEepklUQZriy4A=="}' 2021-07-26T06:34:21.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 362a6d0c-1de4-4cbf-892c-802ffe122833 -i td/osd-markdown/2/new.json 2021-07-26T06:34:21.961 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:34:21.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2021-07-26T06:34:21.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBtV/5gJnslHxAAZa+ojLHeaEepklUQZriy4A== --osd-uuid 362a6d0c-1de4-4cbf-892c-802ffe122833 2021-07-26T06:34:22.005 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:21.984+0000 7ff7cb34cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:22.021 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:22.000+0000 7ff7cb34cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:22.023 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:22.004+0000 7ff7cb34cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:22.024 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:22.004+0000 7ff7cb34cf40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:34:22.024 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:22.004+0000 7ff7cb34cf40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:34:22.024 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:22.004+0000 7ff7cb34cf40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:34:22.077 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:22.056+0000 7ff7cb34cf40 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-07-26T06:34:27.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-07-26T06:34:27.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:34:27.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:34:27.997 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:34:27.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:34:28.480 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:34:28.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:34:28.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:34:28.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:34:28.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:34:28.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:34:28.508 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:28.492+0000 7fa497025f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:28.510 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:28.496+0000 7fa497025f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:28.511 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:28.496+0000 7fa497025f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:34:28.800 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:28.784+0000 7fa497025f40 -1 Falling back to public interface 2021-07-26T06:34:28.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:34:28.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:34:28.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:34:28.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:34:28.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:34:28.819 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:34:28.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:28.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:34:28.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:28.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:34:29.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:30.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:30.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:30.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:34:30.198 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:34:30.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:30.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:34:30.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:31.479 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:34:31.463+0000 7fa497025f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:34:31.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:31.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:31.541 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:34:31.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:34:31.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:34:31.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:32.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:34:33.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:34:33.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:34:33.070 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:34:33.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:34:33.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:34:33.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:34:33.491 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/157627,v1:127.0.0.1:6819/157627] [v2:127.0.0.1:6820/157627,v1:127.0.0.1:6821/157627] exists,up 362a6d0c-1de4-4cbf-892c-802ffe122833 2021-07-26T06:34:33.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:34:33.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:34:33.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:34:33.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2021-07-26T06:34:33.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:34:33.873 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' does not exist 2021-07-26T06:34:33.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:34:33.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:34:34.369 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' already exists 2021-07-26T06:34:34.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:34:35.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:34:37.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2021-07-26T06:34:37.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2021-07-26T06:34:37.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2021-07-26T06:34:37.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-07-26T06:34:37.229 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:34:37.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2021-07-26T06:34:37.383 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:34:37.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2021-07-26T06:34:37.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2021-07-26T06:34:37.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2021-07-26T06:34:37.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-07-26T06:34:37.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2021-07-26T06:34:37.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:34:37.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:34:37.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:34:37.884 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:34:37.884 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:34:37.884 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:34:37.885 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:34:37.885 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:34:37.885 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:34:37.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:34:37.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:34:37.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:34:38.247 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:34:38.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:34:38.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:34:38.699 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:34:38.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:34:48.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:34:48.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:34:48.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:34:49.234 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:34:49.235 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:34:49.236 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:34:49.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:34:49.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:34:49.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:34:49.605 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:34:49.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:34:49.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:34:50.212 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:34:50.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:35:00.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:35:00.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:35:00.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:35:00.723 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:35:00.723 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:35:00.724 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:35:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:35:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:35:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:35:00.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:35:00.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:35:00.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:35:01.119 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:35:01.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:35:01.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:35:01.556 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:35:01.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:35:11.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2021-07-26T06:35:26.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:35:26.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2021-07-26T06:35:26.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2021-07-26T06:35:26.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2021-07-26T06:35:27.070 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:35:27.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-07-26T06:35:27.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:35:27.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:35:27.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:35:27.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:35:27.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:35:27.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:35:27.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:35:27.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:35:27.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:35:27.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:35:27.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:35:27.189 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:35:27.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:35:27.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:35:27.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:35:27.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:27.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:35:27.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:35:27.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:27.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:35:27.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:35:27.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:35:27.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:35:27.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:27.210 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:27.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:35:27.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:35:27.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:35:27.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-07-26T06:35:27.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-07-26T06:35:27.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-07-26T06:35:27.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-07-26T06:35:27.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:35:27.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:35:27.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:35:27.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:35:27.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:35:27.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:35:27.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:35:27.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:35:27.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:35:27.217 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:35:27.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:35:27.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:35:27.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:35:27.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:35:27.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:35:27.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:27.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:35:27.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:35:27.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:35:27.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:35:27.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:35:27.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:35:27.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:35:27.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:27.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:27.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:35:27.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:35:27.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:35:27.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-07-26T06:35:27.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:35:27.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:27.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:27.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153711 2021-07-26T06:35:27.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:35:27.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:35:27.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:35:27.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-07-26T06:35:27.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2021-07-26T06:35:27.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2021-07-26T06:35:27.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2021-07-26T06:35:27.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-07-26T06:35:27.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:35:27.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:35:27.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:35:27.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-07-26T06:35:27.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-07-26T06:35:27.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:35:27.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:27.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:27.615 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:27.616 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:27.616 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:27.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:27.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:35:27.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:35:27.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:35:27.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:35:27.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:35:27.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:35:27.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:35:27.910 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:35:27.910 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:35:27.910 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:35:27.911 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:35:27.911 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:27.912 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:27.912 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:35:27.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:35:27.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get fsid 2021-07-26T06:35:28.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:35:28.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:35:28.004 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:35:28.004 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:35:28.004 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:35:28.006 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:35:28.006 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:35:28.006 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:35:28.007 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:35:28.007 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:28.007 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:28.007 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:35:28.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:35:28.008 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get mon_host 2021-07-26T06:35:28.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2021-07-26T06:35:28.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-07-26T06:35:28.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:35:28.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:35:28.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:35:28.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-07-26T06:35:28.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:35:28.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:35:28.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:28.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:28.996 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:28.996 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:28.997 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:28.997 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:28.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:35:28.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:35:29.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2021-07-26T06:35:29.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:35:29.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:35:29.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:35:29.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:35:29.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-07-26T06:35:29.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:35:29.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:35:29.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:35:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:35:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-07-26T06:35:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-07-26T06:35:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:35:29.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:35:29.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:35:29.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:35:29.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:35:29.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:29.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:29.038 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:29.038 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:29.038 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:29.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:29.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:29.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:35:29.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:35:29.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:35:29.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:35:29.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:35:29.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:35:29.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:35:29.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:29.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:35:29.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:35:29.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2021-07-26T06:35:29.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:35:29.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1e1a48fc-a487-4c24-a8c7-4ff9f171e72e 2021-07-26T06:35:29.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1e1a48fc-a487-4c24-a8c7-4ff9f171e72e' 2021-07-26T06:35:29.044 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 1e1a48fc-a487-4c24-a8c7-4ff9f171e72e 2021-07-26T06:35:29.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:35:29.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxV/5gqUtbAhAAbEBut/FchzV9Urywd+CKrg== 2021-07-26T06:35:29.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxV/5gqUtbAhAAbEBut/FchzV9Urywd+CKrg=="}' 2021-07-26T06:35:29.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1e1a48fc-a487-4c24-a8c7-4ff9f171e72e -i td/osd-markdown/0/new.json 2021-07-26T06:35:29.405 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:35:29.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2021-07-26T06:35:29.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCxV/5gqUtbAhAAbEBut/FchzV9Urywd+CKrg== --osd-uuid 1e1a48fc-a487-4c24-a8c7-4ff9f171e72e 2021-07-26T06:35:29.451 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:29.434+0000 7fb650bd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:29.454 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:29.434+0000 7fb650bd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:29.455 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:29.438+0000 7fb650bd0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:29.456 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:29.438+0000 7fb650bd0f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:35:29.457 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:29.438+0000 7fb650bd0f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:35:29.457 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:29.438+0000 7fb650bd0f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:35:29.539 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:29.522+0000 7fb650bd0f40 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-07-26T06:35:34.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-07-26T06:35:34.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:35:34.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:35:34.946 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:35:34.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:35:35.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:35:35.447 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:35:35.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:35:35.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:35.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:35:35.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:35:35.473 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:35.454+0000 7fed4b67af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:35.475 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:35.458+0000 7fed4b67af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:35.477 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:35.458+0000 7fed4b67af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:35.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:35:35.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:35:35.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:35:35.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:35:35.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:35:35.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:35.793 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:35:35.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:35:35.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:35.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:36.032 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:36.014+0000 7fed4b67af40 -1 Falling back to public interface 2021-07-26T06:35:36.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:37.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:37.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:37.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:35:37.134 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:35:37.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:37.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:37.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:38.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:38.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:38.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:35:38.515 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:35:38.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:38.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:38.572 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:38.554+0000 7fed4b67af40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:35:38.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:39.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:39.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:39.931 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:35:39.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:35:39.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:39.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:35:40.327 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/159788,v1:127.0.0.1:6803/159788] [v2:127.0.0.1:6804/159788,v1:127.0.0.1:6805/159788] exists,up 1e1a48fc-a487-4c24-a8c7-4ff9f171e72e 2021-07-26T06:35:40.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:35:40.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:35:40.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:35:40.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2021-07-26T06:35:40.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:35:40.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:35:40.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:35:40.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:35:40.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-07-26T06:35:40.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:35:40.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:35:40.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:35:40.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:35:40.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-07-26T06:35:40.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-07-26T06:35:40.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:35:40.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:35:40.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:35:40.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:35:40.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:35:40.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:40.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:40.338 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:40.338 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:40.338 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:40.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:40.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:40.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:35:40.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:35:40.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:35:40.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:35:40.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:35:40.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:35:40.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:35:40.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:40.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:35:40.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:35:40.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2021-07-26T06:35:40.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:35:40.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1fdc54b9-ca76-441e-a374-7b69b8bc309f 2021-07-26T06:35:40.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1fdc54b9-ca76-441e-a374-7b69b8bc309f' 2021-07-26T06:35:40.344 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 1fdc54b9-ca76-441e-a374-7b69b8bc309f 2021-07-26T06:35:40.345 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:35:40.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8V/5gzdYqFBAAF/FFS+eWeE0xfylxExMSsA== 2021-07-26T06:35:40.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8V/5gzdYqFBAAF/FFS+eWeE0xfylxExMSsA=="}' 2021-07-26T06:35:40.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1fdc54b9-ca76-441e-a374-7b69b8bc309f -i td/osd-markdown/1/new.json 2021-07-26T06:35:40.838 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:35:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2021-07-26T06:35:40.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC8V/5gzdYqFBAAF/FFS+eWeE0xfylxExMSsA== --osd-uuid 1fdc54b9-ca76-441e-a374-7b69b8bc309f 2021-07-26T06:35:40.886 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:40.870+0000 7f1aee4ebf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:40.889 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:40.870+0000 7f1aee4ebf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:40.890 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:40.874+0000 7f1aee4ebf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:40.891 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:40.874+0000 7f1aee4ebf40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:35:40.892 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:40.874+0000 7f1aee4ebf40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:35:40.892 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:40.874+0000 7f1aee4ebf40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:35:40.959 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:40.942+0000 7f1aee4ebf40 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-07-26T06:35:46.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-07-26T06:35:46.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:35:46.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:35:46.771 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:35:46.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:35:47.256 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:35:47.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:35:47.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:47.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:35:47.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:35:47.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:35:47.283 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:47.266+0000 7fd63b386f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:47.305 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:47.286+0000 7fd63b386f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:47.307 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:47.290+0000 7fd63b386f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:47.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:35:47.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:35:47.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:35:47.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:35:47.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:35:47.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:47.635 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:35:47.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:35:47.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:47.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:47.845 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:47.826+0000 7fd63b386f40 -1 Falling back to public interface 2021-07-26T06:35:48.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:49.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:49.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:49.018 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:35:49.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:35:49.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:49.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:49.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:50.375 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:50.357+0000 7fd63b386f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:35:50.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:50.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:50.428 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:35:50.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:35:50.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:50.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:50.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:35:51.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:35:51.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:51.988 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:35:51.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:35:51.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:51.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:35:52.359 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/161194,v1:127.0.0.1:6811/161194] [v2:127.0.0.1:6812/161194,v1:127.0.0.1:6813/161194] exists,up 1fdc54b9-ca76-441e-a374-7b69b8bc309f 2021-07-26T06:35:52.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:35:52.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:35:52.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:35:52.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2021-07-26T06:35:52.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:35:52.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:35:52.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:35:52.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:35:52.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-07-26T06:35:52.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:35:52.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:35:52.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:35:52.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:35:52.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-07-26T06:35:52.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-07-26T06:35:52.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:35:52.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:35:52.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:35:52.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:35:52.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:35:52.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:35:52.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:35:52.367 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:35:52.368 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:35:52.368 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:35:52.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:52.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:35:52.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:35:52.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:35:52.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:35:52.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:35:52.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:35:52.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:35:52.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:35:52.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:52.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:35:52.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:35:52.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2021-07-26T06:35:52.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:35:52.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8772a895-47b8-4545-b602-3911166c351f 2021-07-26T06:35:52.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8772a895-47b8-4545-b602-3911166c351f' 2021-07-26T06:35:52.372 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 8772a895-47b8-4545-b602-3911166c351f 2021-07-26T06:35:52.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:35:52.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIV/5g9b/8FRAAXVautzaL4iJRcDna/ah7zg== 2021-07-26T06:35:52.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIV/5g9b/8FRAAXVautzaL4iJRcDna/ah7zg=="}' 2021-07-26T06:35:52.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8772a895-47b8-4545-b602-3911166c351f -i td/osd-markdown/2/new.json 2021-07-26T06:35:52.881 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:35:52.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2021-07-26T06:35:52.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDIV/5g9b/8FRAAXVautzaL4iJRcDna/ah7zg== --osd-uuid 8772a895-47b8-4545-b602-3911166c351f 2021-07-26T06:35:52.927 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:52.909+0000 7fba77538f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:52.930 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:52.909+0000 7fba77538f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:52.931 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:52.913+0000 7fba77538f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:52.932 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:52.913+0000 7fba77538f40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:35:52.933 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:52.913+0000 7fba77538f40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:35:52.933 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:52.913+0000 7fba77538f40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:35:53.030 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:53.009+0000 7fba77538f40 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-07-26T06:35:58.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-07-26T06:35:58.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:35:58.937 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:35:58.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:35:58.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:35:59.398 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:35:59.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:35:59.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:35:59.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:35:59.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:35:59.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:35:59.426 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:59.405+0000 7ffb6501af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:59.428 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:59.409+0000 7ffb6501af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:59.429 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:59.409+0000 7ffb6501af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:35:59.736 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:35:59.717+0000 7ffb6501af40 -1 Falling back to public interface 2021-07-26T06:35:59.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:35:59.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:35:59.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:35:59.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:35:59.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:35:59.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:35:59.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:35:59.752 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:35:59.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:35:59.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:00.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:01.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:01.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:01.123 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:36:01.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:36:01.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:01.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:01.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:02.192 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:36:02.174+0000 7ffb6501af40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:36:02.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:02.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:02.500 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:36:02.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:36:02.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:02.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:02.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:36:03.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:36:03.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:36:03.895 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:36:03.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:36:03.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:36:03.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:36:04.261 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/162600,v1:127.0.0.1:6819/162600] [v2:127.0.0.1:6820/162600,v1:127.0.0.1:6821/162600] exists,up 8772a895-47b8-4545-b602-3911166c351f 2021-07-26T06:36:04.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:36:04.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:36:04.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:36:04.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2021-07-26T06:36:04.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:36:04.654 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' does not exist 2021-07-26T06:36:04.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:36:04.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:36:05.175 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' already exists 2021-07-26T06:36:05.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:36:06.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:36:08.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2021-07-26T06:36:08.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2021-07-26T06:36:08.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2021-07-26T06:36:08.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-07-26T06:36:08.226 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:36:08.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2021-07-26T06:36:08.371 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:36:08.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2021-07-26T06:36:08.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-07-26T06:36:08.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2021-07-26T06:36:08.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-07-26T06:36:08.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-07-26T06:36:08.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:36:08.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:36:08.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:36:08.858 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:36:08.858 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:36:08.859 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:36:08.859 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:08.859 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:36:08.859 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:36:08.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:36:08.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:36:08.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:36:09.249 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:09.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:36:09.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:36:09.701 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:36:09.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:36:19.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:36:19.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:36:19.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:36:20.197 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:36:20.198 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:36:20.198 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:36:20.198 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:20.198 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:36:20.199 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:36:20.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:36:20.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:36:20.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:36:20.579 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:20.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:36:20.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:36:21.033 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:36:21.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:36:31.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:36:31.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:36:31.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:36:31.647 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:36:31.648 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:36:31.648 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:36:31.648 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:31.648 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:36:31.649 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:36:31.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:36:31.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:36:31.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:36:32.027 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:32.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:36:32.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:36:32.468 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:36:32.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:36:42.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:36:42.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:36:42.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:36:42.979 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:36:42.980 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:36:42.980 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:36:42.980 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:42.981 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:36:42.981 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:36:42.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:36:42.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:36:42.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:36:43.377 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:36:43.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:36:43.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:36:43.849 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:36:43.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:36:53.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2021-07-26T06:37:08.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:37:09.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2021-07-26T06:37:09.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2021-07-26T06:37:09.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2021-07-26T06:37:09.426 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:37:09.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-07-26T06:37:09.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:37:09.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:37:09.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:37:09.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:37:09.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:37:09.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:37:09.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:37:09.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:37:09.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:37:09.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:37:09.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:37:09.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:37:09.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:37:09.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:37:09.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:37:09.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:09.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:37:09.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:37:09.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:09.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:37:09.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:37:09.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:37:09.770 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:37:09.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:09.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:09.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:37:09.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:37:09.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:37:09.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-07-26T06:37:09.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-07-26T06:37:09.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-07-26T06:37:09.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-07-26T06:37:09.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:37:09.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:37:09.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:37:09.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:37:09.776 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:37:09.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:37:09.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:37:09.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:37:09.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:37:09.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:37:09.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:37:09.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:37:09.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:37:09.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:37:09.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:37:09.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:09.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:37:09.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:37:09.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:37:09.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:37:09.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:37:09.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:37:09.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:37:09.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:09.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:09.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:37:09.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:37:09.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:37:09.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-07-26T06:37:09.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:37:09.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:09.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:09.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153711 2021-07-26T06:37:09.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:37:09.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:37:09.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:37:09.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-07-26T06:37:09.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2021-07-26T06:37:09.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2021-07-26T06:37:09.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2021-07-26T06:37:09.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-07-26T06:37:09.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:37:09.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:37:09.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:37:09.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-07-26T06:37:09.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-07-26T06:37:10.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:37:10.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:10.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:10.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:10.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:10.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:10.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:10.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:37:10.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:37:10.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:37:10.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:37:10.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:37:10.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:37:10.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:37:10.472 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:37:10.472 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:37:10.473 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:37:10.473 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:10.473 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:10.473 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:10.474 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:37:10.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:37:10.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get fsid 2021-07-26T06:37:10.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:37:10.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:37:10.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:37:10.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:37:10.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:37:10.563 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:37:10.563 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:37:10.563 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:37:10.564 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:37:10.564 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:10.564 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:10.564 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:37:10.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:37:10.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get mon_host 2021-07-26T06:37:10.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2021-07-26T06:37:10.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-07-26T06:37:10.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:37:10.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:37:10.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:37:10.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-07-26T06:37:10.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:37:11.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:37:11.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:11.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:11.546 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:11.546 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:11.547 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:11.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:11.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:37:11.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:37:11.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2021-07-26T06:37:11.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:37:11.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:37:11.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:37:11.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:37:11.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-07-26T06:37:11.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:37:11.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:11.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:11.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:11.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-07-26T06:37:11.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-07-26T06:37:11.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:11.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:37:11.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:37:11.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:37:11.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:37:11.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:11.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:11.588 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:11.588 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:11.588 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:11.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:11.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:11.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:11.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:37:11.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:37:11.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:37:11.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:37:11.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:11.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:11.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:11.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:37:11.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:37:11.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2021-07-26T06:37:11.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:37:11.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2bd95d65-dcd8-42cf-9271-7e352dfa7885 2021-07-26T06:37:11.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2bd95d65-dcd8-42cf-9271-7e352dfa7885' 2021-07-26T06:37:11.593 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 2bd95d65-dcd8-42cf-9271-7e352dfa7885 2021-07-26T06:37:11.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:37:11.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXWP5gyO9PIxAApp33aMQdnrsI0566pLnQhg== 2021-07-26T06:37:11.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXWP5gyO9PIxAApp33aMQdnrsI0566pLnQhg=="}' 2021-07-26T06:37:11.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2bd95d65-dcd8-42cf-9271-7e352dfa7885 -i td/osd-markdown/0/new.json 2021-07-26T06:37:11.970 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:37:11.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2021-07-26T06:37:11.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAXWP5gyO9PIxAApp33aMQdnrsI0566pLnQhg== --osd-uuid 2bd95d65-dcd8-42cf-9271-7e352dfa7885 2021-07-26T06:37:12.005 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:11.992+0000 7fa587d6cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:12.009 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:11.996+0000 7fa587d6cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:12.010 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:11.996+0000 7fa587d6cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:12.011 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:12.000+0000 7fa587d6cf40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:37:12.011 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:12.000+0000 7fa587d6cf40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:37:12.012 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:12.000+0000 7fa587d6cf40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:37:12.081 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:12.068+0000 7fa587d6cf40 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-07-26T06:37:17.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-07-26T06:37:17.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:37:17.475 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:37:17.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:37:17.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:37:17.951 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:37:17.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:37:17.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:17.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:37:17.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:37:17.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:37:17.978 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:17.964+0000 7fd52c95cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:17.981 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:17.964+0000 7fd52c95cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:17.982 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:17.968+0000 7fd52c95cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:18.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:37:18.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:37:18.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:37:18.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:37:18.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:37:18.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:18.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:37:18.278 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:37:18.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:18.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:18.512 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:18.496+0000 7fd52c95cf40 -1 Falling back to public interface 2021-07-26T06:37:18.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:19.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:19.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:19.631 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:37:19.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:37:19.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:19.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:19.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:20.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:20.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:20.991 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:37:20.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:37:20.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:20.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:21.002 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:20.988+0000 7fd52c95cf40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:37:21.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:22.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:22.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:22.362 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:37:22.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:37:22.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:22.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:37:22.773 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/164860,v1:127.0.0.1:6803/164860] [v2:127.0.0.1:6804/164860,v1:127.0.0.1:6805/164860] exists,up 2bd95d65-dcd8-42cf-9271-7e352dfa7885 2021-07-26T06:37:22.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:37:22.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:37:22.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:37:22.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2021-07-26T06:37:22.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:37:22.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:37:22.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:37:22.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:37:22.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-07-26T06:37:22.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:37:22.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:22.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:22.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:22.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-07-26T06:37:22.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-07-26T06:37:22.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:22.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:37:22.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:37:22.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:37:22.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:37:22.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:22.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:22.783 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:22.783 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:22.783 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:22.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:22.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:22.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:22.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:37:22.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:37:22.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:37:22.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:37:22.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:22.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:22.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:22.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:37:22.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:37:22.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2021-07-26T06:37:22.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:37:22.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef5af006-19b7-448c-8f7e-5d151517b13b 2021-07-26T06:37:22.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ef5af006-19b7-448c-8f7e-5d151517b13b' 2021-07-26T06:37:22.789 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 ef5af006-19b7-448c-8f7e-5d151517b13b 2021-07-26T06:37:22.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:37:22.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiWP5gSj0WLxAAtVcuGUcnbfnVtBj0HIBaBw== 2021-07-26T06:37:22.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiWP5gSj0WLxAAtVcuGUcnbfnVtBj0HIBaBw=="}' 2021-07-26T06:37:22.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef5af006-19b7-448c-8f7e-5d151517b13b -i td/osd-markdown/1/new.json 2021-07-26T06:37:23.281 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:37:23.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2021-07-26T06:37:23.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAiWP5gSj0WLxAAtVcuGUcnbfnVtBj0HIBaBw== --osd-uuid ef5af006-19b7-448c-8f7e-5d151517b13b 2021-07-26T06:37:23.327 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:23.312+0000 7fa2f20d5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:23.329 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:23.312+0000 7fa2f20d5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:23.331 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:23.316+0000 7fa2f20d5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:23.332 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:23.316+0000 7fa2f20d5f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:37:23.332 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:23.316+0000 7fa2f20d5f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:37:23.332 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:23.316+0000 7fa2f20d5f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:37:23.411 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:23.400+0000 7fa2f20d5f40 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-07-26T06:37:29.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-07-26T06:37:29.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:37:29.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:37:29.226 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:37:29.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:37:29.729 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:37:29.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:37:29.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:29.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:37:29.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:37:29.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:37:29.757 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:29.740+0000 7f4857aeaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:29.760 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:29.744+0000 7f4857aeaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:29.761 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:29.744+0000 7f4857aeaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:30.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:37:30.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:37:30.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:37:30.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:37:30.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:37:30.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:30.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:37:30.120 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:37:30.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:30.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:37:30.296 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:30.280+0000 7f4857aeaf40 -1 Falling back to public interface 2021-07-26T06:37:30.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:31.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:31.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:31.455 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:37:31.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:37:31.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:31.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:37:31.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:32.825 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:32.808+0000 7f4857aeaf40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:37:32.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:32.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:32.841 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:37:32.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:37:32.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:32.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:37:33.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:34.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:34.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:34.403 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:37:34.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:37:34.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:34.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:37:34.761 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/166266,v1:127.0.0.1:6811/166266] [v2:127.0.0.1:6812/166266,v1:127.0.0.1:6813/166266] exists,up ef5af006-19b7-448c-8f7e-5d151517b13b 2021-07-26T06:37:34.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:37:34.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:37:34.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:37:34.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2021-07-26T06:37:34.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:37:34.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:37:34.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:37:34.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:37:34.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-07-26T06:37:34.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:37:34.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:37:34.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:37:34.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:37:34.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-07-26T06:37:34.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-07-26T06:37:34.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:37:34.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:37:34.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:37:34.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:37:34.769 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:37:34.769 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:37:34.770 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:37:34.770 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:37:34.770 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:37:34.770 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:37:34.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:34.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:37:34.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:37:34.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:37:34.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:37:34.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:37:34.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:37:34.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:37:34.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:37:34.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:34.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:37:34.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:37:34.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2021-07-26T06:37:34.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:37:34.774 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 6373a404-56fb-4cdf-9002-8f30f7567d33 2021-07-26T06:37:34.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6373a404-56fb-4cdf-9002-8f30f7567d33 2021-07-26T06:37:34.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6373a404-56fb-4cdf-9002-8f30f7567d33' 2021-07-26T06:37:34.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:37:34.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuWP5g7PQ4LhAApXLAlwJr+0BIR6h1D3a2tw== 2021-07-26T06:37:34.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuWP5g7PQ4LhAApXLAlwJr+0BIR6h1D3a2tw=="}' 2021-07-26T06:37:34.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6373a404-56fb-4cdf-9002-8f30f7567d33 -i td/osd-markdown/2/new.json 2021-07-26T06:37:35.308 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:37:35.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2021-07-26T06:37:35.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAuWP5g7PQ4LhAApXLAlwJr+0BIR6h1D3a2tw== --osd-uuid 6373a404-56fb-4cdf-9002-8f30f7567d33 2021-07-26T06:37:35.352 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:35.336+0000 7fa929c46f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:35.380 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:35.364+0000 7fa929c46f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:35.382 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:35.368+0000 7fa929c46f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:35.383 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:35.368+0000 7fa929c46f40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:37:35.384 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:35.368+0000 7fa929c46f40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:37:35.384 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:35.368+0000 7fa929c46f40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:37:35.481 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:35.464+0000 7fa929c46f40 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-07-26T06:37:41.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-07-26T06:37:41.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:37:41.337 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:37:41.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:37:41.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:37:41.838 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:37:41.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:37:41.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:37:41.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:37:41.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:37:41.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:37:41.866 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:41.852+0000 7f9625930f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:41.893 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:41.876+0000 7f9625930f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:41.895 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:41.880+0000 7f9625930f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:37:42.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:37:42.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:37:42.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:37:42.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:37:42.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:37:42.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:42.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:37:42.228 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:37:42.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:42.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:37:42.428 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:42.412+0000 7f9625930f40 -1 Falling back to public interface 2021-07-26T06:37:42.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:43.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:43.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:43.615 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:37:43.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:37:43.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:43.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:37:44.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:44.919 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:37:44.904+0000 7f9625930f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:37:45.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:45.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:45.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:37:45.036 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:37:45.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:45.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:37:45.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:37:46.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:37:46.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:37:46.490 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:37:46.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:37:46.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:37:46.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:37:46.854 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/167673,v1:127.0.0.1:6819/167673] [v2:127.0.0.1:6820/167673,v1:127.0.0.1:6821/167673] exists,up 6373a404-56fb-4cdf-9002-8f30f7567d33 2021-07-26T06:37:46.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:37:46.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:37:46.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:37:46.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2021-07-26T06:37:46.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-07-26T06:37:47.231 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' does not exist 2021-07-26T06:37:47.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2021-07-26T06:37:47.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2021-07-26T06:37:48.153 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'rbd' already exists 2021-07-26T06:37:48.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:37:49.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2021-07-26T06:37:50.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2021-07-26T06:37:50.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2021-07-26T06:37:50.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2021-07-26T06:37:50.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-07-26T06:37:50.523 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:37:50.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2021-07-26T06:37:50.664 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T06:37:50.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2021-07-26T06:37:50.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-07-26T06:37:50.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2021-07-26T06:37:50.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-07-26T06:37:50.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-07-26T06:37:50.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:37:50.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:37:50.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:37:51.157 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:37:51.158 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:37:51.158 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:37:51.158 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:37:51.159 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:37:51.159 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:37:51.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:37:51.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:37:51.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:37:51.544 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:37:51.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:37:51.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:37:51.977 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:37:51.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:38:01.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:38:02.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:38:02.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:38:02.472 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:38:02.473 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:38:02.473 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:38:02.473 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:38:02.474 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:38:02.474 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:38:02.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:38:02.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:38:02.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:38:02.858 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:38:02.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:38:02.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:38:03.311 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:38:03.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:38:13.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:38:13.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:38:13.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:38:13.839 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:38:13.840 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:38:13.840 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:38:13.840 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:38:13.840 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:38:13.841 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:38:13.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:38:13.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:38:13.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:38:14.210 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:38:14.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:38:14.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:38:14.602 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:38:14.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:38:24.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-07-26T06:38:24.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-07-26T06:38:24.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-07-26T06:38:25.101 INFO:tasks.workunit.client.0.smithi094.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-07-26T06:38:25.102 INFO:tasks.workunit.client.0.smithi094.stdout:-1 0.29306 root default 2021-07-26T06:38:25.102 INFO:tasks.workunit.client.0.smithi094.stdout:-3 0.29306 host smithi094 2021-07-26T06:38:25.102 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:38:25.102 INFO:tasks.workunit.client.0.smithi094.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-07-26T06:38:25.103 INFO:tasks.workunit.client.0.smithi094.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-07-26T06:38:25.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-07-26T06:38:25.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-07-26T06:38:25.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-07-26T06:38:25.464 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:38:25.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-07-26T06:38:25.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-07-26T06:38:25.845 INFO:tasks.workunit.client.0.smithi094.stderr:marked down osd.0. 2021-07-26T06:38:25.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-07-26T06:38:35.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2021-07-26T06:38:35.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2021-07-26T06:38:35.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2021-07-26T06:38:36.216 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-07-26T06:38:36.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-07-26T06:38:36.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:38:36.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:38:36.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:38:36.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:38:36.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:38:36.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:38:36.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:38:36.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:38:36.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:38:36.334 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:38:36.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:38:36.335 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:38:36.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:38:36.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:38:36.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:38:36.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:36.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:38:36.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:38:36.341 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:36.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:38:36.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:38:36.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:38:36.355 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:38:36.356 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:36.356 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:36.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:38:36.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:38:36.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:38:36.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-07-26T06:38:36.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-07-26T06:38:36.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-07-26T06:38:36.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-07-26T06:38:36.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:38:36.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:38:36.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:38:36.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:38:36.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:38:36.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:38:36.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:38:36.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:38:36.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:38:36.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:38:36.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:38:36.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:38:36.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:38:36.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:38:36.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:38:36.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:36.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:38:36.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:38:36.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:38:36.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:38:36.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:38:36.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:38:36.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:38:36.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:36.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:36.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:38:36.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:38:36.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:38:36.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-07-26T06:38:36.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:38:36.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:36.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:36.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153711 2021-07-26T06:38:36.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:38:36.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:38:36.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:38:36.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-07-26T06:38:36.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2021-07-26T06:38:36.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2021-07-26T06:38:36.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2021-07-26T06:38:36.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-07-26T06:38:36.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:38:36.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:38:36.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:38:36.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-07-26T06:38:36.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-07-26T06:38:36.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:38:36.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:36.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:36.757 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:36.758 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:36.758 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:36.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:38:36.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:38:37.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:38:37.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:38:37.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:38:37.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:38:37.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:38:37.050 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:38:37.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:38:37.050 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:38:37.051 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:38:37.051 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:38:37.051 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:37.051 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:37.052 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:38:37.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:38:37.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get fsid 2021-07-26T06:38:37.150 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:38:37.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:38:37.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:38:37.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:38:37.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:38:37.153 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:38:37.153 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:38:37.153 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:38:37.154 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:38:37.154 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:37.154 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:37.154 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153711/ceph-mon.a.asok 2021-07-26T06:38:37.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:38:37.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.153711/ceph-mon.a.asok config get mon_host 2021-07-26T06:38:37.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2021-07-26T06:38:37.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2021-07-26T06:38:37.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:38:37.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:38:37.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:38:37.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2021-07-26T06:38:37.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:38:38.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:38:38.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:38.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:38.134 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:38.134 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:38.134 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:38.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:38:38.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:38:38.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:38:38.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2021-07-26T06:38:38.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:38:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:38:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:38:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:38:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2021-07-26T06:38:38.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:38:38.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:38:38.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:38:38.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:38:38.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-07-26T06:38:38.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-07-26T06:38:38.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:38:38.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:38:38.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:38:38.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:38:38.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:38:38.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:38.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:38.174 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:38.174 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:38.174 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:38.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:38:38.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:38:38.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:38:38.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:38:38.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:38:38.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:38:38.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:38:38.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:38:38.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:38:38.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:38.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:38:38.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:38:38.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2021-07-26T06:38:38.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:38:38.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c1f7dda-4602-4665-bcd4-33546ca70102 2021-07-26T06:38:38.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2c1f7dda-4602-4665-bcd4-33546ca70102' 2021-07-26T06:38:38.181 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 2c1f7dda-4602-4665-bcd4-33546ca70102 2021-07-26T06:38:38.182 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:38:38.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuWP5g40z5ChAA4rEe/gN5RIoVbqQXhEwZVw== 2021-07-26T06:38:38.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuWP5g40z5ChAA4rEe/gN5RIoVbqQXhEwZVw=="}' 2021-07-26T06:38:38.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c1f7dda-4602-4665-bcd4-33546ca70102 -i td/osd-markdown/0/new.json 2021-07-26T06:38:38.533 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:38:38.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2021-07-26T06:38:38.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBuWP5g40z5ChAA4rEe/gN5RIoVbqQXhEwZVw== --osd-uuid 2c1f7dda-4602-4665-bcd4-33546ca70102 2021-07-26T06:38:38.576 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:38.565+0000 7f9f77a14f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:38.579 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:38.565+0000 7f9f77a14f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:38.580 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:38.569+0000 7f9f77a14f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:38.581 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:38.569+0000 7f9f77a14f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:38:38.581 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:38.569+0000 7f9f77a14f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:38:38.582 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:38.569+0000 7f9f77a14f40 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-07-26T06:38:38.664 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:38.653+0000 7f9f77a14f40 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-07-26T06:38:43.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-07-26T06:38:43.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:38:43.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:38:43.993 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:38:43.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:38:44.450 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:38:44.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:38:44.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:44.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:38:44.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:38:44.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:38:44.478 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:44.465+0000 7f5205852f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:44.507 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:44.497+0000 7f5205852f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:44.509 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:44.497+0000 7f5205852f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:44.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:38:44.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:38:44.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:38:44.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:38:44.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:38:44.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:44.827 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:38:44.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:38:44.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:44.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:38:45.058 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:45.045+0000 7f5205852f40 -1 Falling back to public interface 2021-07-26T06:38:45.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:46.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:46.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:46.231 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:38:46.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:38:46.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:46.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:38:46.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:47.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:47.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:47.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:38:47.572 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:38:47.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:47.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:38:47.584 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:47.573+0000 7f5205852f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:38:47.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:48.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:48.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:48.935 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:38:48.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:38:48.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:48.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:38:49.340 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/169905,v1:127.0.0.1:6803/169905] [v2:127.0.0.1:6804/169905,v1:127.0.0.1:6805/169905] exists,up 2c1f7dda-4602-4665-bcd4-33546ca70102 2021-07-26T06:38:49.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:38:49.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:38:49.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:38:49.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2021-07-26T06:38:49.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:38:49.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:38:49.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:38:49.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:38:49.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2021-07-26T06:38:49.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:38:49.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:38:49.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:38:49.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:38:49.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-07-26T06:38:49.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-07-26T06:38:49.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:38:49.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:38:49.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:38:49.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:38:49.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:38:49.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:38:49.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:38:49.349 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:38:49.349 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:38:49.349 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:38:49.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:38:49.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:38:49.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:38:49.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:38:49.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:38:49.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:38:49.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:38:49.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:38:49.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:38:49.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:49.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:38:49.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:38:49.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2021-07-26T06:38:49.354 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:38:49.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97cc375b-31df-46ad-83d5-5da30062fdc7 2021-07-26T06:38:49.355 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 97cc375b-31df-46ad-83d5-5da30062fdc7 2021-07-26T06:38:49.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 97cc375b-31df-46ad-83d5-5da30062fdc7' 2021-07-26T06:38:49.356 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:38:49.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5WP5gYbkfFRAArqPvm/8H+AkuRv7oMB91Kw== 2021-07-26T06:38:49.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5WP5gYbkfFRAArqPvm/8H+AkuRv7oMB91Kw=="}' 2021-07-26T06:38:49.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97cc375b-31df-46ad-83d5-5da30062fdc7 -i td/osd-markdown/1/new.json 2021-07-26T06:38:49.832 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:38:49.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2021-07-26T06:38:49.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB5WP5gYbkfFRAArqPvm/8H+AkuRv7oMB91Kw== --osd-uuid 97cc375b-31df-46ad-83d5-5da30062fdc7 2021-07-26T06:38:49.874 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:49.861+0000 7f9b790f9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:49.877 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:49.865+0000 7f9b790f9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:49.878 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:49.865+0000 7f9b790f9f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:49.879 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:49.865+0000 7f9b790f9f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:38:49.880 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:49.865+0000 7f9b790f9f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:38:49.880 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:49.865+0000 7f9b790f9f40 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-07-26T06:38:49.954 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:49.945+0000 7f9b790f9f40 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-07-26T06:38:55.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-07-26T06:38:55.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:38:55.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:38:55.737 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:38:55.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:38:56.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:38:56.255 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:38:56.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:38:56.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:38:56.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:38:56.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:38:56.283 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:56.269+0000 7f06ddd96f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:56.294 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:56.281+0000 7f06ddd96f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:56.295 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:56.285+0000 7f06ddd96f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:38:56.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:38:56.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:38:56.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:38:56.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:38:56.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:38:56.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:56.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:38:56.631 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:38:56.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:56.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:56.829 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:56.817+0000 7f06ddd96f40 -1 Falling back to public interface 2021-07-26T06:38:56.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:57.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:57.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:57.990 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:38:57.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:38:57.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:57.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:58.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:38:59.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:38:59.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:38:59.362 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:38:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:38:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:38:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:38:59.374 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:38:59.361+0000 7f06ddd96f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:38:59.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:00.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:00.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:00.816 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:39:00.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:00.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:00.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:39:01.169 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/171311,v1:127.0.0.1:6811/171311] [v2:127.0.0.1:6812/171311,v1:127.0.0.1:6813/171311] exists,up 97cc375b-31df-46ad-83d5-5da30062fdc7 2021-07-26T06:39:01.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:01.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:01.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2021-07-26T06:39:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2021-07-26T06:39:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:39:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:39:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:39:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2021-07-26T06:39:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-07-26T06:39:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:01.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:01.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-07-26T06:39:01.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-07-26T06:39:01.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:01.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:01.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:39:01.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-07-26T06:39:01.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:39:01.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:01.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:01.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:01.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:01.180 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:39:01.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:39:01.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' 2021-07-26T06:39:01.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:01.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:39:01.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:39:01.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-07-26T06:39:01.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-07-26T06:39:01.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:01.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:01.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:01.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:39:01.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:39:01.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2021-07-26T06:39:01.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:39:01.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=12b1b2f5-93c1-4fb4-8a8f-daa7f11d7614 2021-07-26T06:39:01.186 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 12b1b2f5-93c1-4fb4-8a8f-daa7f11d7614 2021-07-26T06:39:01.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 12b1b2f5-93c1-4fb4-8a8f-daa7f11d7614' 2021-07-26T06:39:01.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:39:01.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFWP5gHo/lChAA1MsYhX9xHYHGWrrqCTObXQ== 2021-07-26T06:39:01.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFWP5gHo/lChAA1MsYhX9xHYHGWrrqCTObXQ=="}' 2021-07-26T06:39:01.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 12b1b2f5-93c1-4fb4-8a8f-daa7f11d7614 -i td/osd-markdown/2/new.json 2021-07-26T06:39:01.688 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:39:01.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2021-07-26T06:39:01.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCFWP5gHo/lChAA1MsYhX9xHYHGWrrqCTObXQ== --osd-uuid 12b1b2f5-93c1-4fb4-8a8f-daa7f11d7614 2021-07-26T06:39:01.746 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:01.725+0000 7fe6bd63ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:01.760 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:01.749+0000 7fe6bd63ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:01.762 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:01.749+0000 7fe6bd63ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:01.763 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:01.753+0000 7fe6bd63ef40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:39:01.763 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:01.753+0000 7fe6bd63ef40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:39:01.764 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:01.753+0000 7fe6bd63ef40 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-07-26T06:39:01.852 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:01.841+0000 7fe6bd63ef40 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-07-26T06:39:07.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-07-26T06:39:07.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:39:07.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:39:07.634 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:39:07.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:39:08.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:39:08.109 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:39:08.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b76e3569-8161-4fbf-aef2-8cefba4b7546 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.153711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:08.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:39:08.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:39:08.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:39:08.135 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:08.121+0000 7f9a27165f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:08.137 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:08.125+0000 7f9a27165f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:08.138 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:08.125+0000 7f9a27165f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:08.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:39:08.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:39:08.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:39:08.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:39:08.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:39:08.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:08.646 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:39:08.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:39:08.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:08.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:08.694 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:08.681+0000 7f9a27165f40 -1 Falling back to public interface 2021-07-26T06:39:08.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:09.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:09.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:09.961 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:39:09.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:39:09.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:09.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:10.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:10.954 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:10.941+0000 7f9a27165f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:39:11.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:11.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:11.351 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:39:11.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:39:11.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:11.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:11.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:12.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:12.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:12.738 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:39:12.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:12.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:12.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:39:13.094 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/172717,v1:127.0.0.1:6819/172717] [v2:127.0.0.1:6820/172717,v1:127.0.0.1:6821/172717] exists,up 12b1b2f5-93c1-4fb4-8a8f-daa7f11d7614 2021-07-26T06:39:13.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:13.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:13.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:13.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2021-07-26T06:39:13.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=169905 2021-07-26T06:39:13.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 169905 2021-07-26T06:39:13.109 INFO:tasks.workunit.client.0.smithi094.stdout: PID TTY TIME CMD 2021-07-26T06:39:13.109 INFO:tasks.workunit.client.0.smithi094.stdout: 169905 ? 00:00:00 ceph-osd 2021-07-26T06:39:13.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2021-07-26T06:39:13.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2021-07-26T06:39:13.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2021-07-26T06:39:13.489 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-07-26T06:39:13.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2021-07-26T06:39:14.031 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already stopped. 2021-07-26T06:39:14.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2021-07-26T06:39:29.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2021-07-26T06:39:29.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2021-07-26T06:39:29.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2021-07-26T06:39:29.396 INFO:tasks.workunit.client.0.smithi094.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-07-26T06:39:29.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 169905 2021-07-26T06:39:29.408 INFO:tasks.workunit.client.0.smithi094.stdout: PID TTY TIME CMD 2021-07-26T06:39:29.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-07-26T06:39:29.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:39:29.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:39:29.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:39:29.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:29.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:29.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:29.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:29.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:29.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:29.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:39:29.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:39:29.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:39:29.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:39:29.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:39:29.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:39:29.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:29.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:39:29.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:39:29.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:29.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:39:29.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:39:29.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:39:29.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:39:29.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:29.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:39:29.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:39:29.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:39:29.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:39:29.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:39:29.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-markdown 0 2021-07-26T06:39:29.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-07-26T06:39:29.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:39:29.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-07-26T06:39:29.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:29.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:29.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:29.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:29.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:29.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:29.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:39:29.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:39:29.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:39:29.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:39:29.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:39:29.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:39:29.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:29.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:39:29.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:39:29.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:29.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:39:29.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:39:29.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-07-26T06:39:29.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:39:29.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:29.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153711 2021-07-26T06:39:29.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153711 2021-07-26T06:39:29.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:39:29.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:39:29.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:39:29.570 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:39:29.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:39:29.587 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2021-07-26T06:39:29.587 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2021-07-26T06:39:29.648 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:39:29.663 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:39:29.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:39:29.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:39:29.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:39:29.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:39:29.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:39:29.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:39:29.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:39:29.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:39:29.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-reactivate 2021-07-26T06:39:29.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2021-07-26T06:39:29.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2021-07-26T06:39:29.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2021-07-26T06:39:29.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2021-07-26T06:39:29.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2021-07-26T06:39:29.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2021-07-26T06:39:29.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=7c00b291-86e1-4b42-a113-a8bf4550462c --auth-supported=none ' 2021-07-26T06:39:29.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2021-07-26T06:39:29.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2021-07-26T06:39:29.669 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:39:29.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2021-07-26T06:39:29.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2021-07-26T06:39:29.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2021-07-26T06:39:29.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2021-07-26T06:39:29.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2021-07-26T06:39:29.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-07-26T06:39:29.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:39:29.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-07-26T06:39:29.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:29.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:29.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:29.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:29.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:29.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:29.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:39:29.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:39:29.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:39:29.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:39:29.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:39:29.680 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:39:29.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:29.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:39:29.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:39:29.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:29.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:39:29.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:39:29.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-07-26T06:39:29.686 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:39:29.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:29.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:29.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173414 2021-07-26T06:39:29.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:39:29.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:39:29.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2021-07-26T06:39:29.690 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:39:29.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:29.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:29.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173414 2021-07-26T06:39:29.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:39:29.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:39:29.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:39:29.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:39:29.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2021-07-26T06:39:29.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2021-07-26T06:39:29.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2021-07-26T06:39:29.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2021-07-26T06:39:29.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2021-07-26T06:39:29.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:39:29.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:39:29.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:39:29.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2021-07-26T06:39:29.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2021-07-26T06:39:30.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:39:30.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:30.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:30.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:30.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:30.069 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:30.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173414/$cluster-$name.asok' 2021-07-26T06:39:30.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.173414/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:39:30.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:39:30.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:39:30.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:39:30.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:39:30.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:39:30.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:39:30.386 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:39:30.386 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:39:30.386 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:39:30.387 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:39:30.388 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:30.388 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:30.388 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173414/ceph-mon.a.asok 2021-07-26T06:39:30.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:39:30.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.173414/ceph-mon.a.asok config get fsid 2021-07-26T06:39:30.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:39:30.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:39:30.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:39:30.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:39:30.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:39:30.471 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:39:30.472 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:39:30.472 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:39:30.472 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:39:30.472 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:30.473 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:30.473 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173414/ceph-mon.a.asok 2021-07-26T06:39:30.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:39:30.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.173414/ceph-mon.a.asok config get mon_host 2021-07-26T06:39:30.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2021-07-26T06:39:30.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2021-07-26T06:39:30.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:39:30.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:39:30.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:39:30.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2021-07-26T06:39:30.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:39:31.441 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:39:31.441 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:31.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:31.442 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:31.442 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:31.442 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:31.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173414/$cluster-$name.asok' 2021-07-26T06:39:31.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:39:31.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.173414/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:39:31.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2021-07-26T06:39:31.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2021-07-26T06:39:31.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:39:31.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:39:31.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:39:31.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2021-07-26T06:39:31.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7c00b291-86e1-4b42-a113-a8bf4550462c --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-07-26T06:39:31.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:31.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:31.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:31.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-07-26T06:39:31.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-07-26T06:39:31.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:31.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:31.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:39:31.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-07-26T06:39:31.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:39:31.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:31.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:31.482 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:31.482 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:31.482 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:31.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173414/$cluster-$name.asok' 2021-07-26T06:39:31.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173414/$cluster-$name.asok' 2021-07-26T06:39:31.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:31.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:39:31.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:39:31.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-07-26T06:39:31.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-07-26T06:39:31.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:31.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:31.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:31.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:39:31.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:39:31.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2021-07-26T06:39:31.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:39:31.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a029691b-3e0b-4416-94a5-97a26040839e 2021-07-26T06:39:31.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a029691b-3e0b-4416-94a5-97a26040839e' 2021-07-26T06:39:31.487 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 a029691b-3e0b-4416-94a5-97a26040839e 2021-07-26T06:39:31.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:39:31.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjWP5g4R0OHRAAyq+q04boz4zGt+3p2C8+zw== 2021-07-26T06:39:31.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjWP5g4R0OHRAAyq+q04boz4zGt+3p2C8+zw=="}' 2021-07-26T06:39:31.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a029691b-3e0b-4416-94a5-97a26040839e -i td/osd-reactivate/0/new.json 2021-07-26T06:39:31.857 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:39:31.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2021-07-26T06:39:31.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7c00b291-86e1-4b42-a113-a8bf4550462c --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.173414/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCjWP5g4R0OHRAAyq+q04boz4zGt+3p2C8+zw== --osd-uuid a029691b-3e0b-4416-94a5-97a26040839e 2021-07-26T06:39:31.904 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:31.893+0000 7f3e01af8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:31.907 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:31.897+0000 7f3e01af8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:31.907 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:31.897+0000 7f3e01af8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:31.909 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:31.897+0000 7f3e01af8f40 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-07-26T06:39:31.909 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:31.897+0000 7f3e01af8f40 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-07-26T06:39:31.909 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:31.897+0000 7f3e01af8f40 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-07-26T06:39:32.005 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:31.993+0000 7f3e01af8f40 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2021-07-26T06:39:37.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2021-07-26T06:39:37.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:39:37.539 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:39:37.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:39:37.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:39:38.038 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:39:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:39:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7c00b291-86e1-4b42-a113-a8bf4550462c --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.173414/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:39:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:39:38.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:39:38.066 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:38.053+0000 7f986e3baf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:38.068 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:38.057+0000 7f986e3baf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:38.074 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:38.057+0000 7f986e3baf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:38.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:39:38.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:39:38.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:39:38.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:39:38.381 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:39:38.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:39:38.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:38.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:39:38.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:38.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:38.609 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:38.597+0000 7f986e3baf40 -1 Falling back to public interface 2021-07-26T06:39:38.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:39.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:39.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:39.745 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:39:39.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:39:39.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:39.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:40.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:41.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:41.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:41.083 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:39:41.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:39:41.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:41.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:41.140 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:41.129+0000 7f986e3baf40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:39:41.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:42.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:42.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:42.501 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:39:42.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:42.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:42.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:42.866 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174518,v1:127.0.0.1:6803/174518] [v2:127.0.0.1:6804/174518,v1:127.0.0.1:6805/174518] exists,up a029691b-3e0b-4416-94a5-97a26040839e 2021-07-26T06:39:42.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:42.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:42.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:42.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2021-07-26T06:39:42.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:42.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:42.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:42.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:42.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:42.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:42.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2021-07-26T06:39:42.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2021-07-26T06:39:42.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2021-07-26T06:39:42.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2021-07-26T06:39:42.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-reactivate 2021-07-26T06:39:42.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T06:39:42.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T06:39:42.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T06:39:42.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-reactivate/0 2021-07-26T06:39:42.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=7c00b291-86e1-4b42-a113-a8bf4550462c --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-07-26T06:39:42.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:42.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:42.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:42.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-07-26T06:39:42.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-07-26T06:39:42.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:42.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:42.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T06:39:42.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-07-26T06:39:42.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T06:39:42.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:42.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:42.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:42.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:42.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:42.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173414/$cluster-$name.asok' 2021-07-26T06:39:42.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173414/$cluster-$name.asok' 2021-07-26T06:39:42.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:42.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-07-26T06:39:42.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-07-26T06:39:42.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:42.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:42.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:42.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T06:39:42.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T06:39:42.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-reactivate/0 2021-07-26T06:39:42.992 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:39:42.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T06:39:42.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=7c00b291-86e1-4b42-a113-a8bf4550462c --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.173414/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:42.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-reactivate/0/whoami 2021-07-26T06:39:42.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T06:39:42.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T06:39:42.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T06:39:42.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T06:39:43.009 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:42.997+0000 7f30964f1f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:43.012 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:43.001+0000 7f30964f1f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:43.014 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:43.001+0000 7f30964f1f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:43.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T06:39:43.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:39:43.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:39:43.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:39:43.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:39:43.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:43.356 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:39:43.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:39:43.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:43.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:43.557 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:43.545+0000 7f30964f1f40 -1 Falling back to public interface 2021-07-26T06:39:43.718 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174518,v1:127.0.0.1:6803/174518] [v2:127.0.0.1:6804/174518,v1:127.0.0.1:6805/174518] exists,up a029691b-3e0b-4416-94a5-97a26040839e 2021-07-26T06:39:43.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:43.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:43.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:43.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2021-07-26T06:39:43.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-07-26T06:39:43.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:39:43.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-07-26T06:39:43.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:43.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:43.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:43.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:43.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:43.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:43.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:39:43.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:39:43.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:39:43.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:39:43.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:39:43.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:39:43.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:43.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:39:43.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:39:43.841 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:43.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:39:43.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:39:43.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-07-26T06:39:43.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:39:43.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:43.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:43.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173414 2021-07-26T06:39:43.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:39:43.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:39:43.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T06:39:43.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-reactivate 0 2021-07-26T06:39:43.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-07-26T06:39:43.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T06:39:43.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-07-26T06:39:43.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:43.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:43.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:43.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:43.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:43.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:43.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:39:43.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:39:43.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:39:43.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:39:43.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:39:43.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:39:43.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:43.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:39:43.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:39:43.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:43.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:39:43.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T06:39:43.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-07-26T06:39:43.870 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:39:43.870 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:43.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173414 2021-07-26T06:39:43.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173414 2021-07-26T06:39:43.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:39:43.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:39:43.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T06:39:43.875 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T06:39:43.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T06:39:43.893 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2021-07-26T06:39:43.893 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2021-07-26T06:39:43.953 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T06:39:43.969 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T06:39:43.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:39:43.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T06:39:43.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T06:39:43.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T06:39:43.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T06:39:43.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T06:39:43.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T06:39:43.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T06:39:43.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-recovery-prio 2021-07-26T06:39:43.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2021-07-26T06:39:43.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2021-07-26T06:39:43.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-07-26T06:39:43.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-07-26T06:39:43.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2021-07-26T06:39:43.974 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2021-07-26T06:39:43.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none ' 2021-07-26T06:39:43.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:39:43.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: export objects=200 2021-07-26T06:39:43.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: objects=200 2021-07-26T06:39:43.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: export poolprefix=test 2021-07-26T06:39:43.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: poolprefix=test 2021-07-26T06:39:43.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: export FORCE_PRIO=255 2021-07-26T06:39:43.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: FORCE_PRIO=255 2021-07-26T06:39:43.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export NORMAL_PRIO=190 2021-07-26T06:39:43.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: NORMAL_PRIO=190 2021-07-26T06:39:43.976 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: set 2021-07-26T06:39:43.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T06:39:43.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: local 'funcs=TEST_recovery_pool_priority 2021-07-26T06:39:43.978 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_recovery_priority' 2021-07-26T06:39:43.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-07-26T06:39:43.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-07-26T06:39:43.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-07-26T06:39:43.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-07-26T06:39:43.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-07-26T06:39:43.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:39:43.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-07-26T06:39:43.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:39:43.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:39:43.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:39:43.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:39:43.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:39:43.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:39:43.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:39:43.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:39:43.983 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:39:43.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:39:43.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:39:43.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:39:43.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:43.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:39:43.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:39:43.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:39:43.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:39:43.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:39:43.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-07-26T06:39:43.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:39:43.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:43.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:43.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175221 2021-07-26T06:39:43.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:39:43.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:39:43.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-07-26T06:39:43.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:39:43.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:43.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:43.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.175221 2021-07-26T06:39:43.999 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:39:44.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T06:39:44.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T06:39:44.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:39:44.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-07-26T06:39:44.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_pool_priority td/osd-recovery-prio 2021-07-26T06:39:44.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:323: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2021-07-26T06:39:44.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:324: TEST_recovery_pool_priority: local pools=3 2021-07-26T06:39:44.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:325: TEST_recovery_pool_priority: local OSDS=2 2021-07-26T06:39:44.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2021-07-26T06:39:44.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-07-26T06:39:44.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:39:44.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:39:44.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:39:44.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-07-26T06:39:44.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-07-26T06:39:44.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:39:44.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:44.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:44.393 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:44.393 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:44.394 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:44.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:39:44.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:39:44.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:39:44.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:39:44.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:39:44.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:39:44.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:39:44.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:39:44.676 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:39:44.676 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:39:44.676 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:39:44.677 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:39:44.677 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:44.677 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:44.678 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-mon.a.asok 2021-07-26T06:39:44.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:39:44.679 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.175221/ceph-mon.a.asok config get fsid 2021-07-26T06:39:44.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:39:44.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:39:44.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:39:44.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:39:44.763 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:39:44.763 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:39:44.764 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:39:44.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:39:44.764 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:39:44.765 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:44.765 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:44.765 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-mon.a.asok 2021-07-26T06:39:44.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:39:44.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.175221/ceph-mon.a.asok config get mon_host 2021-07-26T06:39:44.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2021-07-26T06:39:44.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2021-07-26T06:39:44.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:39:44.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:39:44.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:39:44.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2021-07-26T06:39:44.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:39:45.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:39:45.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:45.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:45.747 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:45.747 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:45.748 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:45.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:39:45.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:39:45.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:39:45.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:329: TEST_recovery_pool_priority: export CEPH_ARGS 2021-07-26T06:39:45.781 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: expr 2 - 1 2021-07-26T06:39:45.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: seq 0 1 2021-07-26T06:39:45.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:39:45.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2021-07-26T06:39:45.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-07-26T06:39:45.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:39:45.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:39:45.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:39:45.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2021-07-26T06:39:45.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:39:45.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:45.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:45.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:45.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-07-26T06:39:45.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-07-26T06:39:45.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:45.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:45.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:39:45.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-07-26T06:39:45.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:39:45.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:45.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:45.792 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:45.792 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:45.793 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:45.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:39:45.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:39:45.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:45.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:39:45.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:39:45.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-07-26T06:39:45.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-07-26T06:39:45.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:45.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:45.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:45.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:39:45.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:39:45.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2021-07-26T06:39:45.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:39:45.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5ddc045-8052-4cdc-b547-d481d3073a2a 2021-07-26T06:39:45.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a5ddc045-8052-4cdc-b547-d481d3073a2a' 2021-07-26T06:39:45.797 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 a5ddc045-8052-4cdc-b547-d481d3073a2a 2021-07-26T06:39:45.797 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:39:45.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxWP5gEADNLxAAcgv67LNz7ABAF2P5rj5jqw== 2021-07-26T06:39:45.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxWP5gEADNLxAAcgv67LNz7ABAF2P5rj5jqw=="}' 2021-07-26T06:39:45.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5ddc045-8052-4cdc-b547-d481d3073a2a -i td/osd-recovery-prio/0/new.json 2021-07-26T06:39:46.137 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:39:46.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2021-07-26T06:39:46.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCxWP5gEADNLxAAcgv67LNz7ABAF2P5rj5jqw== --osd-uuid a5ddc045-8052-4cdc-b547-d481d3073a2a 2021-07-26T06:39:46.182 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:46.169+0000 7f9b50200f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:46.192 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:46.181+0000 7f9b50200f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:46.193 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:46.181+0000 7f9b50200f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:46.195 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:46.185+0000 7f9b50200f40 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-07-26T06:39:46.195 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:46.185+0000 7f9b50200f40 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-07-26T06:39:46.195 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:46.185+0000 7f9b50200f40 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-07-26T06:39:46.239 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:46.229+0000 7f9b50200f40 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-07-26T06:39:51.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-07-26T06:39:51.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:39:51.854 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:39:51.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:39:51.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:39:52.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:39:52.337 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:39:52.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:52.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:39:52.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:39:52.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:39:52.367 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:52.357+0000 7fc650cc7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:52.370 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:52.357+0000 7fc650cc7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:52.371 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:52.361+0000 7fc650cc7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:52.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:39:52.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:39:52.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:39:52.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:39:52.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:39:52.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:52.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:39:52.674 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:39:52.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:52.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:52.909 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:52.897+0000 7fc650cc7f40 -1 Falling back to public interface 2021-07-26T06:39:53.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:54.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:54.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:54.029 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:39:54.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:39:54.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:54.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:54.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:55.367 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:55.357+0000 7fc650cc7f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:39:55.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:55.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:55.387 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:39:55.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:39:55.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:55.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:55.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:39:56.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:39:56.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:39:56.833 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:39:56.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:39:56.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:39:56.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:39:57.197 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/176327,v1:127.0.0.1:6803/176327] [v2:127.0.0.1:6804/176327,v1:127.0.0.1:6805/176327] exists,up a5ddc045-8052-4cdc-b547-d481d3073a2a 2021-07-26T06:39:57.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:39:57.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:39:57.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:39:57.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:39:57.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2021-07-26T06:39:57.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-07-26T06:39:57.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:39:57.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:39:57.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:39:57.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2021-07-26T06:39:57.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:39:57.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:39:57.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:39:57.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:39:57.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-07-26T06:39:57.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-07-26T06:39:57.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:39:57.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:39:57.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:39:57.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-07-26T06:39:57.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:39:57.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:39:57.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:39:57.206 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:39:57.207 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:39:57.207 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:39:57.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:39:57.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:39:57.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:39:57.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:39:57.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:39:57.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-07-26T06:39:57.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-07-26T06:39:57.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:39:57.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:39:57.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:39:57.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:39:57.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:39:57.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2021-07-26T06:39:57.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:39:57.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8fd10340-224c-4fa2-a83b-98caa34093bc 2021-07-26T06:39:57.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8fd10340-224c-4fa2-a83b-98caa34093bc' 2021-07-26T06:39:57.212 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 8fd10340-224c-4fa2-a83b-98caa34093bc 2021-07-26T06:39:57.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:39:57.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9WP5gPUwMDRAAvarTDiCnN9u7jTpdLKNUJw== 2021-07-26T06:39:57.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9WP5gPUwMDRAAvarTDiCnN9u7jTpdLKNUJw=="}' 2021-07-26T06:39:57.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8fd10340-224c-4fa2-a83b-98caa34093bc -i td/osd-recovery-prio/1/new.json 2021-07-26T06:39:57.697 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:39:57.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2021-07-26T06:39:57.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC9WP5gPUwMDRAAvarTDiCnN9u7jTpdLKNUJw== --osd-uuid 8fd10340-224c-4fa2-a83b-98caa34093bc 2021-07-26T06:39:57.747 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:57.737+0000 7fbb8f8aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:57.749 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:57.737+0000 7fbb8f8aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:57.751 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:57.741+0000 7fbb8f8aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:39:57.752 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:57.741+0000 7fbb8f8aaf40 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-07-26T06:39:57.752 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:57.741+0000 7fbb8f8aaf40 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-07-26T06:39:57.752 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:57.741+0000 7fbb8f8aaf40 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-07-26T06:39:57.811 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:39:57.801+0000 7fbb8f8aaf40 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-07-26T06:40:03.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-07-26T06:40:03.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:40:03.590 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:40:03.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:40:03.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:40:04.110 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:40:04.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:40:04.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:40:04.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:40:04.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:40:04.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:40:04.140 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:40:04.129+0000 7fe06bb40f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:04.142 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:40:04.129+0000 7fe06bb40f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:04.143 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:40:04.133+0000 7fe06bb40f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:40:04.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:40:04.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:40:04.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:40:04.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:40:04.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:40:04.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:04.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:40:04.445 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:40:04.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:04.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:04.689 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:40:04.677+0000 7fe06bb40f40 -1 Falling back to public interface 2021-07-26T06:40:04.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:05.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:05.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:05.809 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:40:05.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:40:05.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:05.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:06.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:07.193 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:40:07.181+0000 7fe06bb40f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:40:07.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:07.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:07.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:40:07.208 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:40:07.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:07.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:07.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:40:08.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:40:08.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:40:08.781 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:40:08.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:40:08.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:40:08.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:40:09.129 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/177733,v1:127.0.0.1:6811/177733] [v2:127.0.0.1:6812/177733,v1:127.0.0.1:6813/177733] exists,up 8fd10340-224c-4fa2-a83b-98caa34093bc 2021-07-26T06:40:09.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:40:09.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:40:09.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:40:09.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: seq 1 3 2021-07-26T06:40:09.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:09.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test1 1 1 2021-07-26T06:40:09.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-07-26T06:40:09.615 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test1' already exists 2021-07-26T06:40:09.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:40:10.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-07-26T06:40:11.367 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 2 2021-07-26T06:40:11.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:11.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test2 1 1 2021-07-26T06:40:11.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-07-26T06:40:12.355 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test2' already exists 2021-07-26T06:40:12.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:40:13.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-07-26T06:40:14.349 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 2 size to 2 2021-07-26T06:40:14.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:14.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test3 1 1 2021-07-26T06:40:14.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-07-26T06:40:14.982 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test3' already exists 2021-07-26T06:40:14.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:40:15.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2021-07-26T06:40:16.783 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 3 size to 2 2021-07-26T06:40:16.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: sleep 5 2021-07-26T06:40:21.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:343: TEST_recovery_pool_priority: wait_for_clean 2021-07-26T06:40:21.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:40:21.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:40:21.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:40:21.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:40:21.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:40:21.800 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:40:21.801 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:40:21.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:40:21.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:40:21.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:40:22.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:40:22.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:40:22.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:40:22.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:40:22.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:40:22.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:40:22.364 INFO:tasks.workunit.client.0.smithi094.stderr:1' 2021-07-26T06:40:22.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:40:22.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:22.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:40:22.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T06:40:22.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T06:40:22.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T06:40:22.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:22.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:40:22.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672965 2021-07-26T06:40:22.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672965 2021-07-26T06:40:22.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2021-07-26T06:40:22.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:22.651 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T06:40:22.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:22.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:40:22.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T06:40:22.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:22.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T06:40:22.656 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836488 2021-07-26T06:40:22.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T06:40:22.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:23.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836488 2021-07-26T06:40:23.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:24.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:40:24.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:24.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T06:40:24.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:24.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672965 2021-07-26T06:40:24.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:24.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:40:24.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672965 2021-07-26T06:40:24.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:24.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672965 2021-07-26T06:40:24.392 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672965 2021-07-26T06:40:24.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-07-26T06:40:24.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:40:24.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672965 2021-07-26T06:40:24.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:40:24.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:24.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:25.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 3 == 0 2021-07-26T06:40:25.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:40:25.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:40:25.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:40:25.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:40:25.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:40:25.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:40:25.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:40:25.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=3 2021-07-26T06:40:25.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:40:25.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:25.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:25.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 3 = 3 2021-07-26T06:40:25.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:40:25.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:40:25.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:345: TEST_recovery_pool_priority: ceph pg dump pgs 2021-07-26T06:40:26.315 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:40:26.316 INFO:tasks.workunit.client.0.smithi094.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:40:17.388585+0000 0'0 28:18 [1,0] 1 [1,0] 1 0'0 2021-07-26T06:40:14.742649+0000 0'0 2021-07-26T06:40:14.742649+0000 0 2021-07-26T06:40:26.316 INFO:tasks.workunit.client.0.smithi094.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:40:14.365784+0000 0'0 27:23 [0,1] 0 [0,1] 0 0'0 2021-07-26T06:40:11.920547+0000 0'0 2021-07-26T06:40:11.920547+0000 0 2021-07-26T06:40:26.316 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:40:11.536231+0000 0'0 27:24 [1,0] 1 [1,0] 1 0'0 2021-07-26T06:40:09.436912+0000 0'0 2021-07-26T06:40:09.436912+0000 0 2021-07-26T06:40:26.317 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:40:26.317 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:40:26.318 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:40:26.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:349: TEST_recovery_pool_priority: local PG1 2021-07-26T06:40:26.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:350: TEST_recovery_pool_priority: local POOLNUM1 2021-07-26T06:40:26.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:351: TEST_recovery_pool_priority: local pool1 2021-07-26T06:40:26.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local chk_osd1_1 2021-07-26T06:40:26.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local chk_osd1_2 2021-07-26T06:40:26.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local PG2 2021-07-26T06:40:26.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local POOLNUM2 2021-07-26T06:40:26.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:357: TEST_recovery_pool_priority: local pool2 2021-07-26T06:40:26.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local chk_osd2_1 2021-07-26T06:40:26.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local chk_osd2_2 2021-07-26T06:40:26.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: seq 1 3 2021-07-26T06:40:26.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:26.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-07-26T06:40:26.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2021-07-26T06:40:26.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:40:26.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=1 2021-07-26T06:40:26.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:40:26.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=0 2021-07-26T06:40:26.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z '' ']' 2021-07-26T06:40:26.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: PG1=1.0 2021-07-26T06:40:26.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: POOLNUM1=1 2021-07-26T06:40:26.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:370: TEST_recovery_pool_priority: pool1=test1 2021-07-26T06:40:26.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: chk_osd1_1=1 2021-07-26T06:40:26.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: chk_osd1_2=0 2021-07-26T06:40:26.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:26.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2021-07-26T06:40:26.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-07-26T06:40:27.057 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:40:27.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=0 2021-07-26T06:40:27.059 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:40:27.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=1 2021-07-26T06:40:27.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2021-07-26T06:40:27.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2021-07-26T06:40:27.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: PG2=2.0 2021-07-26T06:40:27.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: POOLNUM2=2 2021-07-26T06:40:27.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:377: TEST_recovery_pool_priority: pool2=test2 2021-07-26T06:40:27.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: chk_osd2_1=0 2021-07-26T06:40:27.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: chk_osd2_2=1 2021-07-26T06:40:27.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: break 2021-07-26T06:40:27.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2021-07-26T06:40:27.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:385: TEST_recovery_pool_priority: '[' test2 = '' ']' 2021-07-26T06:40:27.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: seq 1 3 2021-07-26T06:40:27.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:27.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2021-07-26T06:40:27.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:27.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2021-07-26T06:40:27.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-07-26T06:40:27.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2021-07-26T06:40:27.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:395: TEST_recovery_pool_priority: delete_pool test3 2021-07-26T06:40:27.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2021-07-26T06:40:27.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-07-26T06:40:27.562 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test3' does not exist 2021-07-26T06:40:27.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:399: TEST_recovery_pool_priority: pool1_extra_prio=1 2021-07-26T06:40:27.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:400: TEST_recovery_pool_priority: pool2_extra_prio=2 2021-07-26T06:40:27.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: expr 190 + 1 2021-07-26T06:40:27.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: pool1_prio=191 2021-07-26T06:40:27.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: expr 190 + 2 2021-07-26T06:40:27.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool2_prio=192 2021-07-26T06:40:27.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-07-26T06:40:28.483 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 1 2021-07-26T06:40:28.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2021-07-26T06:40:29.311 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 recovery_priority to 1 2021-07-26T06:40:29.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:406: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-07-26T06:40:30.233 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 2 size to 1 2021-07-26T06:40:30.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2021-07-26T06:40:31.053 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 2 recovery_priority to 2 2021-07-26T06:40:31.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: wait_for_clean 2021-07-26T06:40:31.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:40:31.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:40:31.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:40:31.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:40:31.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:40:31.067 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:40:31.068 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:40:31.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:40:31.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:40:31.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:40:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:40:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:40:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:40:31.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:40:31.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:40:31.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:40:31.665 INFO:tasks.workunit.client.0.smithi094.stderr:1' 2021-07-26T06:40:31.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:40:31.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:31.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:40:31.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836492 2021-07-26T06:40:31.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836492 2021-07-26T06:40:31.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492' 2021-07-26T06:40:31.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:40:31.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:40:31.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672969 2021-07-26T06:40:31.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672969 2021-07-26T06:40:31.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2021-07-26T06:40:31.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:31.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836492 2021-07-26T06:40:31.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:31.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:40:31.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836492 2021-07-26T06:40:31.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:31.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836492 2021-07-26T06:40:31.989 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836492 2021-07-26T06:40:31.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-07-26T06:40:31.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:32.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836490 -lt 21474836492 2021-07-26T06:40:32.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:33.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:40:33.354 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:33.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836490 -lt 21474836492 2021-07-26T06:40:33.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:40:34.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 299 -eq 0 ']' 2021-07-26T06:40:34.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:40:35.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836492 -lt 21474836492 2021-07-26T06:40:35.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:40:35.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672969 2021-07-26T06:40:35.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:40:35.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:40:35.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672969 2021-07-26T06:40:35.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:40:35.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672969 2021-07-26T06:40:35.138 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672969 2021-07-26T06:40:35.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-07-26T06:40:35.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:40:35.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672970 -lt 42949672969 2021-07-26T06:40:35.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:40:35.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:35.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:35.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 2 == 0 2021-07-26T06:40:35.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:40:35.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:40:35.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:40:35.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:40:35.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:40:35.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:40:35.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:40:36.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T06:40:36.316 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:40:36.316 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:40:36.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:40:36.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 2 2021-07-26T06:40:36.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:40:36.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:40:36.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-07-26T06:40:36.818 INFO:tasks.workunit.client.0.smithi094.stderr:10+0 records in 2021-07-26T06:40:36.818 INFO:tasks.workunit.client.0.smithi094.stderr:10+0 records out 2021-07-26T06:40:36.819 INFO:tasks.workunit.client.0.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0543317 s, 193 MB/s 2021-07-26T06:40:36.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: p=1 2021-07-26T06:40:36.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-07-26T06:40:36.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-07-26T06:40:36.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:36.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-07-26T06:40:37.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:37.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-07-26T06:40:37.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:37.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-07-26T06:40:38.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:38.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-07-26T06:40:38.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:38.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-07-26T06:40:38.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:38.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-07-26T06:40:39.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:39.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-07-26T06:40:39.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:39.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-07-26T06:40:39.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:39.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-07-26T06:40:40.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:40.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-07-26T06:40:41.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:41.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-07-26T06:40:41.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:41.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-07-26T06:40:42.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:42.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-07-26T06:40:42.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:42.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-07-26T06:40:42.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:42.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-07-26T06:40:43.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:43.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-07-26T06:40:44.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:44.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-07-26T06:40:44.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:44.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-07-26T06:40:45.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:45.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-07-26T06:40:45.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:45.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-07-26T06:40:45.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:45.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-07-26T06:40:46.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:46.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-07-26T06:40:47.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:47.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-07-26T06:40:47.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:47.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-07-26T06:40:48.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:48.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-07-26T06:40:48.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:48.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-07-26T06:40:49.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:49.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-07-26T06:40:49.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:49.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-07-26T06:40:50.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:50.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-07-26T06:40:50.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:50.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-07-26T06:40:51.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:51.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-07-26T06:40:51.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:51.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-07-26T06:40:52.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:52.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-07-26T06:40:52.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:52.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-07-26T06:40:53.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:53.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-07-26T06:40:53.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:53.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-07-26T06:40:54.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:54.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-07-26T06:40:54.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:54.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-07-26T06:40:54.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:54.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-07-26T06:40:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-07-26T06:40:55.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:55.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-07-26T06:40:56.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:56.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-07-26T06:40:56.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:56.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-07-26T06:40:57.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:57.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-07-26T06:40:57.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:57.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-07-26T06:40:58.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:58.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-07-26T06:40:58.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:58.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-07-26T06:40:59.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:40:59.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-07-26T06:41:00.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:00.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-07-26T06:41:00.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:00.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-07-26T06:41:01.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:01.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-07-26T06:41:01.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:01.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-07-26T06:41:02.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:02.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-07-26T06:41:02.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:02.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-07-26T06:41:03.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:03.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-07-26T06:41:03.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:03.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-07-26T06:41:04.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:04.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-07-26T06:41:04.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:04.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-07-26T06:41:04.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:04.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-07-26T06:41:05.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:05.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-07-26T06:41:06.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:06.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-07-26T06:41:06.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:06.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-07-26T06:41:06.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:06.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-07-26T06:41:07.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:07.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-07-26T06:41:07.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:07.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-07-26T06:41:08.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:08.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-07-26T06:41:08.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:08.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-07-26T06:41:09.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:09.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-07-26T06:41:09.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:09.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-07-26T06:41:10.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:10.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-07-26T06:41:10.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:10.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-07-26T06:41:11.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:11.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-07-26T06:41:12.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:12.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-07-26T06:41:12.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:12.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-07-26T06:41:13.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:13.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-07-26T06:41:13.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:13.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-07-26T06:41:13.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:13.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-07-26T06:41:14.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:14.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-07-26T06:41:15.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:15.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-07-26T06:41:15.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:15.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-07-26T06:41:15.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:15.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-07-26T06:41:16.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:16.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-07-26T06:41:16.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:16.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-07-26T06:41:17.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:17.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-07-26T06:41:17.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:17.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-07-26T06:41:18.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:18.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-07-26T06:41:18.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:18.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-07-26T06:41:19.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:19.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-07-26T06:41:19.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:19.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-07-26T06:41:19.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:19.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-07-26T06:41:20.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:20.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-07-26T06:41:21.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:21.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-07-26T06:41:21.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:21.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-07-26T06:41:21.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:21.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-07-26T06:41:22.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-07-26T06:41:23.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:23.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-07-26T06:41:24.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:24.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-07-26T06:41:25.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:25.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-07-26T06:41:25.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:25.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-07-26T06:41:25.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:25.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-07-26T06:41:26.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:26.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-07-26T06:41:26.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:26.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-07-26T06:41:27.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:27.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-07-26T06:41:28.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:28.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-07-26T06:41:28.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:28.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-07-26T06:41:29.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:29.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-07-26T06:41:30.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:30.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-07-26T06:41:30.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:30.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-07-26T06:41:30.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:30.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-07-26T06:41:31.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:31.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-07-26T06:41:31.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:31.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-07-26T06:41:31.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:31.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-07-26T06:41:32.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:32.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-07-26T06:41:33.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:33.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-07-26T06:41:33.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:33.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-07-26T06:41:33.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:33.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-07-26T06:41:34.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:34.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-07-26T06:41:34.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:34.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-07-26T06:41:36.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:36.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-07-26T06:41:36.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:36.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-07-26T06:41:37.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:37.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-07-26T06:41:37.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:37.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-07-26T06:41:37.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:37.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-07-26T06:41:38.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:38.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-07-26T06:41:39.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:39.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-07-26T06:41:39.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:39.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-07-26T06:41:39.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:39.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-07-26T06:41:40.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:40.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-07-26T06:41:40.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:40.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-07-26T06:41:41.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:41.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-07-26T06:41:41.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:41.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-07-26T06:41:42.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:42.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-07-26T06:41:42.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:42.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-07-26T06:41:43.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:43.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-07-26T06:41:43.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:43.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-07-26T06:41:43.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:43.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-07-26T06:41:44.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:44.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-07-26T06:41:45.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:45.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-07-26T06:41:45.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:45.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-07-26T06:41:46.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:46.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-07-26T06:41:46.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:46.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-07-26T06:41:47.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:47.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-07-26T06:41:48.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:48.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-07-26T06:41:48.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:48.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-07-26T06:41:48.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:48.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-07-26T06:41:49.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:49.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-07-26T06:41:49.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:49.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-07-26T06:41:50.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:50.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-07-26T06:41:51.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:51.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-07-26T06:41:51.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:51.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-07-26T06:41:52.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:52.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-07-26T06:41:52.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:52.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-07-26T06:41:52.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:52.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-07-26T06:41:53.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:53.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-07-26T06:41:54.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:54.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-07-26T06:41:54.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:54.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-07-26T06:41:54.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:54.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-07-26T06:41:55.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:55.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-07-26T06:41:55.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:55.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-07-26T06:41:56.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:56.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-07-26T06:41:56.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:56.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-07-26T06:41:57.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:57.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-07-26T06:41:57.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:57.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-07-26T06:41:58.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:58.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-07-26T06:41:58.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:58.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-07-26T06:41:59.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:41:59.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-07-26T06:42:00.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:00.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-07-26T06:42:00.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:00.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-07-26T06:42:01.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:01.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-07-26T06:42:01.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:01.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-07-26T06:42:01.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:01.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-07-26T06:42:02.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:02.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-07-26T06:42:03.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:03.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-07-26T06:42:03.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:03.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-07-26T06:42:03.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:03.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-07-26T06:42:04.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:04.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-07-26T06:42:04.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:04.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-07-26T06:42:05.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:05.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-07-26T06:42:05.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:05.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-07-26T06:42:06.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:06.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-07-26T06:42:06.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:06.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-07-26T06:42:07.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:07.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-07-26T06:42:07.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:07.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-07-26T06:42:07.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:07.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-07-26T06:42:08.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:08.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-07-26T06:42:08.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:08.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-07-26T06:42:09.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:09.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-07-26T06:42:09.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:09.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-07-26T06:42:10.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:10.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-07-26T06:42:11.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:11.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-07-26T06:42:11.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:11.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-07-26T06:42:12.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:12.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-07-26T06:42:12.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:12.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-07-26T06:42:13.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:13.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-07-26T06:42:13.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:13.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-07-26T06:42:14.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:14.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-07-26T06:42:14.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:14.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-07-26T06:42:15.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:15.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-07-26T06:42:15.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:15.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-07-26T06:42:16.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:16.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-07-26T06:42:16.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 1 + 1 2021-07-26T06:42:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=2 2021-07-26T06:42:16.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-07-26T06:42:16.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-07-26T06:42:16.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:16.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2021-07-26T06:42:17.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:17.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2021-07-26T06:42:17.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:17.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2021-07-26T06:42:17.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:17.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2021-07-26T06:42:18.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:18.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2021-07-26T06:42:18.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:18.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2021-07-26T06:42:19.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:19.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2021-07-26T06:42:19.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:19.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2021-07-26T06:42:20.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:20.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2021-07-26T06:42:21.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:21.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2021-07-26T06:42:21.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:21.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2021-07-26T06:42:21.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:21.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2021-07-26T06:42:22.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:22.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2021-07-26T06:42:22.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:22.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2021-07-26T06:42:23.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:23.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2021-07-26T06:42:23.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:23.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2021-07-26T06:42:24.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:24.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2021-07-26T06:42:24.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:24.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2021-07-26T06:42:25.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:25.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2021-07-26T06:42:25.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:25.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2021-07-26T06:42:25.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:25.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2021-07-26T06:42:26.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:26.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2021-07-26T06:42:27.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:27.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2021-07-26T06:42:27.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:27.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2021-07-26T06:42:28.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:28.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2021-07-26T06:42:28.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:28.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2021-07-26T06:42:29.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:29.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2021-07-26T06:42:30.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:30.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2021-07-26T06:42:30.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:30.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2021-07-26T06:42:30.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:30.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2021-07-26T06:42:31.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:31.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2021-07-26T06:42:31.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:31.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2021-07-26T06:42:32.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:32.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2021-07-26T06:42:32.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:32.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2021-07-26T06:42:33.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:33.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2021-07-26T06:42:33.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:33.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2021-07-26T06:42:34.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:34.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2021-07-26T06:42:34.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:34.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2021-07-26T06:42:34.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:34.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2021-07-26T06:42:35.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:35.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2021-07-26T06:42:36.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:36.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2021-07-26T06:42:36.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:36.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2021-07-26T06:42:37.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:37.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2021-07-26T06:42:37.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:37.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2021-07-26T06:42:37.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:37.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2021-07-26T06:42:38.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:38.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2021-07-26T06:42:39.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:39.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2021-07-26T06:42:39.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:39.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2021-07-26T06:42:40.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:40.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2021-07-26T06:42:40.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:40.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2021-07-26T06:42:40.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:40.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2021-07-26T06:42:41.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:41.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2021-07-26T06:42:42.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:42.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2021-07-26T06:42:42.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:42.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2021-07-26T06:42:42.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:42.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2021-07-26T06:42:43.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:43.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2021-07-26T06:42:43.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:43.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2021-07-26T06:42:44.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:44.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2021-07-26T06:42:45.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:45.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2021-07-26T06:42:45.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:45.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2021-07-26T06:42:45.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:45.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2021-07-26T06:42:46.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:46.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2021-07-26T06:42:46.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:46.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2021-07-26T06:42:47.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:47.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2021-07-26T06:42:47.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:47.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2021-07-26T06:42:48.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:48.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2021-07-26T06:42:48.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:48.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2021-07-26T06:42:49.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:49.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2021-07-26T06:42:49.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:49.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2021-07-26T06:42:49.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:49.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2021-07-26T06:42:50.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:50.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2021-07-26T06:42:51.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:51.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2021-07-26T06:42:52.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:52.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2021-07-26T06:42:52.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:52.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2021-07-26T06:42:53.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:53.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2021-07-26T06:42:53.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:53.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2021-07-26T06:42:54.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:54.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2021-07-26T06:42:54.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:54.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2021-07-26T06:42:55.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:55.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2021-07-26T06:42:55.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:55.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2021-07-26T06:42:55.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:55.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2021-07-26T06:42:56.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:56.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2021-07-26T06:42:57.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:57.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2021-07-26T06:42:57.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:57.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2021-07-26T06:42:58.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:58.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2021-07-26T06:42:58.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:58.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2021-07-26T06:42:58.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:58.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2021-07-26T06:42:59.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:59.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2021-07-26T06:42:59.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:42:59.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2021-07-26T06:43:00.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:00.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2021-07-26T06:43:00.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:00.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2021-07-26T06:43:01.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:01.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2021-07-26T06:43:01.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:01.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2021-07-26T06:43:02.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:02.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2021-07-26T06:43:03.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:03.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2021-07-26T06:43:04.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:04.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2021-07-26T06:43:05.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:05.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2021-07-26T06:43:05.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:05.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2021-07-26T06:43:06.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:06.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2021-07-26T06:43:06.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:06.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2021-07-26T06:43:06.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:06.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2021-07-26T06:43:07.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:07.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2021-07-26T06:43:07.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:07.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2021-07-26T06:43:08.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:08.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2021-07-26T06:43:09.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:09.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2021-07-26T06:43:09.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:09.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2021-07-26T06:43:10.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:10.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2021-07-26T06:43:10.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:10.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2021-07-26T06:43:11.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:11.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2021-07-26T06:43:11.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:11.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2021-07-26T06:43:12.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:12.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2021-07-26T06:43:12.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:12.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2021-07-26T06:43:13.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:13.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2021-07-26T06:43:13.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:13.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2021-07-26T06:43:13.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:13.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2021-07-26T06:43:14.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:14.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2021-07-26T06:43:14.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:14.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2021-07-26T06:43:15.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:15.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2021-07-26T06:43:16.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:16.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2021-07-26T06:43:16.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:16.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2021-07-26T06:43:16.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:16.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2021-07-26T06:43:17.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:17.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2021-07-26T06:43:17.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:17.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2021-07-26T06:43:18.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:18.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2021-07-26T06:43:19.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:19.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2021-07-26T06:43:19.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:19.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2021-07-26T06:43:19.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:19.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2021-07-26T06:43:20.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:20.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2021-07-26T06:43:20.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:20.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2021-07-26T06:43:21.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:21.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2021-07-26T06:43:21.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:21.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2021-07-26T06:43:22.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:22.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2021-07-26T06:43:22.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:22.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2021-07-26T06:43:23.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:23.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2021-07-26T06:43:23.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:23.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2021-07-26T06:43:24.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:24.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2021-07-26T06:43:24.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:24.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2021-07-26T06:43:25.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:25.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2021-07-26T06:43:25.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:25.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2021-07-26T06:43:26.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:26.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2021-07-26T06:43:26.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:26.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2021-07-26T06:43:27.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:27.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2021-07-26T06:43:28.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:28.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2021-07-26T06:43:28.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:28.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2021-07-26T06:43:29.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:29.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2021-07-26T06:43:29.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:29.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2021-07-26T06:43:29.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:29.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2021-07-26T06:43:30.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:30.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2021-07-26T06:43:31.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:31.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2021-07-26T06:43:31.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:31.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2021-07-26T06:43:31.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:31.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2021-07-26T06:43:32.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:32.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2021-07-26T06:43:32.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:32.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2021-07-26T06:43:33.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:33.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2021-07-26T06:43:34.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:34.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2021-07-26T06:43:34.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:34.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2021-07-26T06:43:34.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:34.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2021-07-26T06:43:35.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:35.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2021-07-26T06:43:35.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:35.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2021-07-26T06:43:36.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:36.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2021-07-26T06:43:36.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:36.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2021-07-26T06:43:37.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:37.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2021-07-26T06:43:37.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:37.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2021-07-26T06:43:37.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:37.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2021-07-26T06:43:38.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:38.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2021-07-26T06:43:39.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:39.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2021-07-26T06:43:40.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:40.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2021-07-26T06:43:40.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:40.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2021-07-26T06:43:40.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:40.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2021-07-26T06:43:41.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:41.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2021-07-26T06:43:41.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:41.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2021-07-26T06:43:42.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:42.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2021-07-26T06:43:43.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:43.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2021-07-26T06:43:43.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:43.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2021-07-26T06:43:43.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:43.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2021-07-26T06:43:44.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:44.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2021-07-26T06:43:44.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:44.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2021-07-26T06:43:44.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:44.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2021-07-26T06:43:45.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:45.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2021-07-26T06:43:46.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:46.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2021-07-26T06:43:46.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:46.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2021-07-26T06:43:47.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:47.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2021-07-26T06:43:47.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:47.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2021-07-26T06:43:47.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:47.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2021-07-26T06:43:48.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:48.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2021-07-26T06:43:49.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:49.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2021-07-26T06:43:49.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:49.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2021-07-26T06:43:49.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:49.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2021-07-26T06:43:50.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:50.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2021-07-26T06:43:51.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:51.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2021-07-26T06:43:52.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:52.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2021-07-26T06:43:53.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:53.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2021-07-26T06:43:53.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:53.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2021-07-26T06:43:54.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:54.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2021-07-26T06:43:54.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:54.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2021-07-26T06:43:55.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:55.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2021-07-26T06:43:55.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:55.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2021-07-26T06:43:56.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:56.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2021-07-26T06:43:56.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:56.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2021-07-26T06:43:56.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-07-26T06:43:56.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2021-07-26T06:43:57.282 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 2 + 1 2021-07-26T06:43:57.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=3 2021-07-26T06:43:57.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2021-07-26T06:43:57.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test1 2021-07-26T06:43:57.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1-p1 2021-07-26T06:43:57.286 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test1 obj1-p1 2021-07-26T06:43:57.286 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test1 2021-07-26T06:43:57.287 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1-p1 2021-07-26T06:43:57.287 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test1 obj1-p1 2021-07-26T06:43:57.287 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:43:57.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:43:57.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-07-26T06:43:57.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:43:58.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: local otherosd=null 2021-07-26T06:43:58.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:423: TEST_recovery_pool_priority: ceph pg dump pgs 2021-07-26T06:43:58.749 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:43:58.749 INFO:tasks.workunit.client.0.smithi094.stdout:2.0 199 0 0 0 0 2080374784 0 0 595 595 active+clean 2021-07-26T06:40:30.619717+0000 39'595 39:634 [0] 0 [0] 0 0'0 2021-07-26T06:40:11.920547+0000 0'0 2021-07-26T06:40:11.920547+0000 0 2021-07-26T06:43:58.750 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T06:40:28.879573+0000 39'600 39:638 [1] 1 [1] 1 0'0 2021-07-26T06:40:09.436912+0000 0'0 2021-07-26T06:40:09.436912+0000 0 2021-07-26T06:43:58.750 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:43:58.750 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:43:58.751 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:43:58.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: ERRORS=0 2021-07-26T06:43:58.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-07-26T06:43:59.415 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 2 2021-07-26T06:43:59.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-07-26T06:44:00.345 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 2 size to 2 2021-07-26T06:44:00.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph pg dump pgs 2021-07-26T06:44:00.699 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:44:00.699 INFO:tasks.workunit.client.0.smithi094.stdout:2.0 199 0 0 0 0 2080374784 0 0 595 595 active+clean 2021-07-26T06:40:30.619717+0000 39'595 39:634 [0] 0 [0] 0 0'0 2021-07-26T06:40:11.920547+0000 0'0 2021-07-26T06:40:11.920547+0000 0 2021-07-26T06:44:00.699 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T06:40:28.879573+0000 39'600 39:638 [1] 1 [1] 1 0'0 2021-07-26T06:40:09.436912+0000 0'0 2021-07-26T06:40:09.436912+0000 0 2021-07-26T06:44:00.700 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:44:00.700 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:44:00.702 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:44:00.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: set -o pipefail 2021-07-26T06:44:00.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:434: TEST_recovery_pool_priority: count=0 2021-07-26T06:44:00.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-07-26T06:44:00.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-07-26T06:44:00.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-07-26T06:44:00.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-07-26T06:44:01.084 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:44:01.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-07-26T06:44:01.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-07-26T06:44:03.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 0 -eq 10 2021-07-26T06:44:03.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 0 + 1 2021-07-26T06:44:03.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=1 2021-07-26T06:44:03.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-07-26T06:44:03.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-07-26T06:44:03.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-07-26T06:44:03.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-07-26T06:44:03.452 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:44:03.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 0 == 2 2021-07-26T06:44:03.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-07-26T06:44:05.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 1 -eq 10 2021-07-26T06:44:05.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 1 + 1 2021-07-26T06:44:05.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=2 2021-07-26T06:44:05.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-07-26T06:44:05.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-07-26T06:44:05.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-07-26T06:44:05.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-07-26T06:44:05.814 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:44:05.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 1 == 2 2021-07-26T06:44:05.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: sleep 2 2021-07-26T06:44:07.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: test 2 -eq 10 2021-07-26T06:44:07.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: expr 2 + 1 2021-07-26T06:44:07.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:448: TEST_recovery_pool_priority: count=3 2021-07-26T06:44:07.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:435: TEST_recovery_pool_priority: true 2021-07-26T06:44:07.837 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2021-07-26T06:44:07.837 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2021-07-26T06:44:07.837 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:439: TEST_recovery_pool_priority: grep -c true 2021-07-26T06:44:08.191 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:44:08.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: test 2 == 2 2021-07-26T06:44:08.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:440: TEST_recovery_pool_priority: break 2021-07-26T06:44:08.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:450: TEST_recovery_pool_priority: set +o pipefail 2021-07-26T06:44:08.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: ceph pg dump pgs 2021-07-26T06:44:08.825 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:44:08.826 INFO:tasks.workunit.client.0.smithi094.stdout:2.0 200 0 199 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-07-26T06:44:04.431005+0000 39'600 47:655 [0,1] 0 [0] 0 0'0 2021-07-26T06:40:11.920547+0000 0'0 2021-07-26T06:40:11.920547+0000 0 2021-07-26T06:44:08.826 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 200 0 197 0 0 2097152000 0 0 600 600 active+recovering+undersized+degraded+remapped 2021-07-26T06:44:00.514775+0000 39'600 47:660 [1,0] 1 [1] 1 0'0 2021-07-26T06:40:09.436912+0000 0'0 2021-07-26T06:40:09.436912+0000 0 2021-07-26T06:44:08.827 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:44:08.827 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:44:08.828 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:44:08.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: get_asok_path osd.1 2021-07-26T06:44:08.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:44:08.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:44:08.843 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:44:08.844 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:44:08.844 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:44:08.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:44:08.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: CEPH_ARGS= 2021-07-26T06:44:08.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:44:08.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: echo osd.1 2021-07-26T06:44:08.938 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 2021-07-26T06:44:08.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:455: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-07-26T06:44:08.939 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:44:08.939 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:44:08.940 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:44:08.940 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:44:08.940 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:44:08.940 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:44:08.941 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:44:08.941 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:44:08.941 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 191, 2021-07-26T06:44:08.941 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:44:08.942 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:44:08.942 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:44:08.942 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:44:08.942 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:44:08.942 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:44:08.943 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:44:08.943 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:44:08.943 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:44:08.943 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:44:08.944 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "2.0", 2021-07-26T06:44:08.944 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 192, 2021-07-26T06:44:08.944 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:44:08.944 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:44:08.944 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:44:08.945 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:44:08.945 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:44:08.946 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.0 2021-07-26T06:44:08.946 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-07-26T06:44:08.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-07-26T06:44:08.947 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:44:08.947 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:44:08.947 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:44:08.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.0.asok 2021-07-26T06:44:08.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2021-07-26T06:44:08.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.0.asok dump_recovery_reservations 2021-07-26T06:44:09.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.0 2021-07-26T06:44:09.030 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 2021-07-26T06:44:09.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-07-26T06:44:09.031 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:44:09.032 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:44:09.032 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:44:09.032 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:44:09.033 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:44:09.033 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:44:09.033 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:44:09.034 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "2.0", 2021-07-26T06:44:09.034 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 192, 2021-07-26T06:44:09.034 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:44:09.034 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:44:09.035 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:44:09.035 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:44:09.035 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:44:09.035 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:44:09.036 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:44:09.036 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:44:09.036 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:44:09.036 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:44:09.036 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:44:09.037 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 191, 2021-07-26T06:44:09.037 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:44:09.037 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:44:09.038 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:44:09.038 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:44:09.038 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:44:09.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-07-26T06:44:09.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-07-26T06:44:09.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-07-26T06:44:09.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: ITEM=1.0 2021-07-26T06:44:09.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:462: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-07-26T06:44:09.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-07-26T06:44:09.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-07-26T06:44:09.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: PRIO=191 2021-07-26T06:44:09.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:468: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-07-26T06:44:09.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-07-26T06:44:09.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-07-26T06:44:09.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-07-26T06:44:09.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:476: TEST_recovery_pool_priority: ITEM=1.0 2021-07-26T06:44:09.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:477: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-07-26T06:44:09.163 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-07-26T06:44:09.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-07-26T06:44:09.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: PRIO=191 2021-07-26T06:44:09.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:483: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-07-26T06:44:09.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-07-26T06:44:09.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-07-26T06:44:09.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-07-26T06:44:09.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:491: TEST_recovery_pool_priority: ITEM=2.0 2021-07-26T06:44:09.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:492: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-07-26T06:44:09.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-07-26T06:44:09.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-07-26T06:44:09.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:497: TEST_recovery_pool_priority: PRIO=192 2021-07-26T06:44:09.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:498: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-07-26T06:44:09.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-07-26T06:44:09.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-07-26T06:44:09.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-07-26T06:44:09.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: ITEM=2.0 2021-07-26T06:44:09.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:507: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-07-26T06:44:09.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-07-26T06:44:09.341 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-07-26T06:44:09.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:512: TEST_recovery_pool_priority: PRIO=192 2021-07-26T06:44:09.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:513: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-07-26T06:44:09.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:520: TEST_recovery_pool_priority: wait_for_clean 2021-07-26T06:44:09.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:44:09.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:44:09.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:44:09.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:44:09.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:44:09.384 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:44:09.384 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:44:09.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:44:09.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:44:09.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:44:09.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:44:09.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:44:09.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:44:09.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:44:09.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:44:10.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:44:10.196 INFO:tasks.workunit.client.0.smithi094.stderr:1' 2021-07-26T06:44:10.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:44:10.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:10.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:44:10.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836537 2021-07-26T06:44:10.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836537 2021-07-26T06:44:10.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836537' 2021-07-26T06:44:10.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:44:10.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:44:10.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673015 2021-07-26T06:44:10.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673015 2021-07-26T06:44:10.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836537 1-42949673015' 2021-07-26T06:44:10.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:10.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836537 2021-07-26T06:44:10.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:10.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:44:10.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836537 2021-07-26T06:44:10.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:10.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836537 2021-07-26T06:44:10.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2021-07-26T06:44:10.572 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836537 2021-07-26T06:44:10.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:11.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836535 -lt 21474836537 2021-07-26T06:44:11.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:44:12.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:44:12.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:44:13.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836538 -lt 21474836537 2021-07-26T06:44:13.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:44:13.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673015 2021-07-26T06:44:13.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:44:13.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:44:13.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673015 2021-07-26T06:44:13.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:44:13.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673015 2021-07-26T06:44:13.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2021-07-26T06:44:13.025 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949673015 2021-07-26T06:44:13.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:44:13.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673015 -lt 42949673015 2021-07-26T06:44:13.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:44:13.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:13.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:14.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 2 == 0 2021-07-26T06:44:14.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:14.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:14.201 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:14.201 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:14.201 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:14.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:14.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:14.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:14.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:14.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:14.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:15.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:15.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:44:15.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:44:15.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:44:15.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:15.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:15.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:15.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:15.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:15.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:15.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:15.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:15.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:15.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:15.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:15.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:15.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:15.613 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:16.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:16.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:16.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:16.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:16.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:16.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:16.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:16.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:16.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:17.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14850091 2021-07-26T06:44:17.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14850091 '!=' null 2021-07-26T06:44:17.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:17.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:17.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:17.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:17.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:17.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:17.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:17.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:17.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:17.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:17.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:17.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:17.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:17.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:17.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:18.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:18.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:18.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:18.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:18.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:18.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:18.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:18.169 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:18.169 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:19.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344559 2021-07-26T06:44:19.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344559 '!=' null 2021-07-26T06:44:19.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:19.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:19.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:19.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:19.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:19.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:19.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:19.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:19.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:19.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:19.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:19.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:19.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:19.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:19.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:20.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:20.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:20.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:20.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:20.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:20.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:20.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:20.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:20.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:20.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=22712584 2021-07-26T06:44:20.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 22712584 '!=' null 2021-07-26T06:44:20.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:20.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:20.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:20.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:20.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:20.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:20.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:20.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:20.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:20.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:20.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:21.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:21.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:21.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:21.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:22.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:22.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:22.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:22.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:22.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:22.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:22.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:22.329 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:22.329 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:23.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218623 2021-07-26T06:44:23.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218623 '!=' null 2021-07-26T06:44:23.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:23.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:23.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:23.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:23.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:23.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:23.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:23.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:23.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:23.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:23.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:23.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:23.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:23.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:23.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:24.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:24.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:24.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:24.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:24.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:24.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:24.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:24.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:24.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:25.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218938 2021-07-26T06:44:25.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218938 '!=' null 2021-07-26T06:44:25.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:25.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:25.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:25.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:25.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:25.290 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:25.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:25.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:25.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:25.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:25.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:25.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:25.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:25.644 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:25.644 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:26.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:26.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:26.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:26.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:26.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:26.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:26.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:26.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:26.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:26.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16598388 2021-07-26T06:44:26.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16598388 '!=' null 2021-07-26T06:44:26.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:26.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:26.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:26.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:26.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:26.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:26.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:26.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:26.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:26.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:26.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:27.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:27.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:27.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:27.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:27.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:27.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:27.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:27.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:27.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:27.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:27.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:27.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:27.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:28.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16598388 2021-07-26T06:44:28.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16598388 '!=' null 2021-07-26T06:44:28.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:28.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:28.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:28.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:28.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:28.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:28.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:28.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:28.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:28.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:28.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:28.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:28.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:28.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:28.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:29.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:29.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:29.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:29.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:29.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:29.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:29.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:29.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:29.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:30.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092773 2021-07-26T06:44:30.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092773 '!=' null 2021-07-26T06:44:30.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:30.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:30.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:30.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:30.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:30.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:30.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:30.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:30.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:30.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:30.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:31.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:31.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:31.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:31.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:31.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:31.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:31.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:31.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:31.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:31.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:31.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:31.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:31.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:32.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091831 2021-07-26T06:44:32.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091831 '!=' null 2021-07-26T06:44:32.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:32.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:32.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:32.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:32.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:32.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:32.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:32.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:32.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:32.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:32.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:33.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:33.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:33.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:33.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:33.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:33.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:33.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:33.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:33.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:33.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:33.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:33.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:33.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:34.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21838669 2021-07-26T06:44:34.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21838669 '!=' null 2021-07-26T06:44:34.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:34.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:34.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:34.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:34.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:34.639 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:34.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:34.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:34.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:34.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:34.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:35.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:35.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:35.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:35.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:35.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:35.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:35.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:35.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:35.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:35.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:35.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:35.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:35.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:36.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21838669 2021-07-26T06:44:36.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21838669 '!=' null 2021-07-26T06:44:36.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:36.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:36.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:36.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:36.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:36.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:36.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:36.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:36.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:36.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:36.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:36.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:36.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:36.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:36.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:37.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:37.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:37.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:37.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:37.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:37.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:37.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:37.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:37.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:38.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597296 2021-07-26T06:44:38.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597296 '!=' null 2021-07-26T06:44:38.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:38.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:38.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:38.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:38.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:38.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:38.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:38.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:38.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:38.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:38.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:38.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:38.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:38.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:38.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:39.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:39.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:39.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:39.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:39.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:39.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:39.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:39.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:39.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:39.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091214 2021-07-26T06:44:39.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091214 '!=' null 2021-07-26T06:44:39.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:39.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:39.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:39.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:39.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:39.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:39.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:39.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:39.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:39.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:39.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:40.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:40.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:40.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:40.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:41.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:41.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:41.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:41.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:41.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:41.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:41.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20964924 2021-07-26T06:44:41.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20964924 '!=' null 2021-07-26T06:44:41.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:41.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:41.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:42.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:42.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:42.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:42.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:42.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:42.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:42.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:42.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:42.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:42.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:42.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:42.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:43.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:43.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:43.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:43.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:43.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:43.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:43.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:43.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:43.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:44.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218172 2021-07-26T06:44:44.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218172 '!=' null 2021-07-26T06:44:44.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:44.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:44.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:44.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:44.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:44.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:44.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:44.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:44.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:44.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:44.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:44.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:44.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:44.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:44.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:45.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:45.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:45.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:45.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:45.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:45.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:45.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:45.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:45.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:46.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092052 2021-07-26T06:44:46.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092052 '!=' null 2021-07-26T06:44:46.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:46.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:46.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:46.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:46.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:46.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:46.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:46.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:46.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:46.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:46.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:47.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:47.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:47.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:47.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:47.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:47.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:47.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:47.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:47.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:47.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:47.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:47.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:47.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:48.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15724244 2021-07-26T06:44:48.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15724244 '!=' null 2021-07-26T06:44:48.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:48.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:48.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:48.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:48.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:48.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:48.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:48.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:48.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:48.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:48.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:49.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:49.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:49.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:49.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:49.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:49.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:49.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:49.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:49.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:49.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:49.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:49.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:49.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:50.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965715 2021-07-26T06:44:50.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965715 '!=' null 2021-07-26T06:44:50.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:50.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:50.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:50.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:50.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:50.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:50.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:50.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:50.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:50.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:50.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:51.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:51.161 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:51.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:51.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:51.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:51.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:51.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:51.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:51.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:51.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:51.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:51.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:51.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:52.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092063 2021-07-26T06:44:52.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092063 '!=' null 2021-07-26T06:44:52.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:52.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:52.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:52.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:52.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:52.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:52.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:52.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:52.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:52.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:52.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:53.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:53.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:53.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:53.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:53.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:53.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:53.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:53.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:53.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:53.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:53.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:53.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:53.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:54.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092063 2021-07-26T06:44:54.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092063 '!=' null 2021-07-26T06:44:54.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:54.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:54.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:54.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:54.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:54.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:54.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:54.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:54.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:54.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:54.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:54.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:54.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:54.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:54.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:55.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:55.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:55.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:55.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:55.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:55.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:55.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:55.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:55.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:56.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597723 2021-07-26T06:44:56.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597723 '!=' null 2021-07-26T06:44:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:56.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:56.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:56.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:56.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:56.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:56.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:56.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:56.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:57.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:57.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:57.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:57.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:57.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:57.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:57.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:57.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:57.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:57.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:57.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:57.570 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:57.570 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:44:58.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597723 2021-07-26T06:44:58.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597723 '!=' null 2021-07-26T06:44:58.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:44:58.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:44:58.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:44:58.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:44:58.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:44:58.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:44:58.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:44:58.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:44:58.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:44:58.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:44:58.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:44:59.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:44:59.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:44:59.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:44:59.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:44:59.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:44:59.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:44:59.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:44:59.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:44:59.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:44:59.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:44:59.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:44:59.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:44:59.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:00.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092226 2021-07-26T06:45:00.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092226 '!=' null 2021-07-26T06:45:00.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:00.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:00.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:00.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:00.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:00.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:00.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:00.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:00.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:00.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:00.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:00.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:00.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:00.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:00.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:01.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:01.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:01.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:01.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:01.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:01.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:01.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:01.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:01.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:02.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092226 2021-07-26T06:45:02.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092226 '!=' null 2021-07-26T06:45:02.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:02.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:02.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:02.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:02.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:02.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:02.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:02.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:02.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:02.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:02.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:03.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:03.166 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:03.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:03.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:03.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:03.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:03.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:03.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:03.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:03.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:03.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:03.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:03.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:04.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345212 2021-07-26T06:45:04.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345212 '!=' null 2021-07-26T06:45:04.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:04.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:04.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:04.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:04.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:04.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:04.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:04.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:04.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:04.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:04.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:05.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:05.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:05.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:05.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:05.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:05.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:05.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:05.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:05.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:05.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:05.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:05.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:05.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:06.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092248 2021-07-26T06:45:06.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092248 '!=' null 2021-07-26T06:45:06.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:06.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:06.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:06.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:06.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:06.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:06.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:06.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:06.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:06.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:06.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:06.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:06.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:06.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:06.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:07.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:07.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:07.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:07.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:07.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:07.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:07.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:07.814 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:07.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:08.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218991 2021-07-26T06:45:08.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218991 '!=' null 2021-07-26T06:45:08.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:08.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:08.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:08.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:08.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:08.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:08.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:08.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:08.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:08.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:08.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:09.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:09.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:09.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:09.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:09.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:09.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:09.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:09.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:09.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:09.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:09.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:09.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:09.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:10.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092139 2021-07-26T06:45:10.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092139 '!=' null 2021-07-26T06:45:10.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:10.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:10.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:10.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:10.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:10.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:10.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:10.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:10.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:10.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:10.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:10.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:10.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:10.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:10.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:11.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:11.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:11.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:11.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:11.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:11.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:11.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:11.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:11.196 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:11.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092139 2021-07-26T06:45:11.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092139 '!=' null 2021-07-26T06:45:11.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:11.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:11.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:11.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:11.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:11.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:11.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:11.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:11.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:11.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:11.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:12.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:12.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:12.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:12.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:12.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:12.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:12.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:12.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:12.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:12.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:12.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:12.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:12.829 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:13.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092234 2021-07-26T06:45:13.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092234 '!=' null 2021-07-26T06:45:13.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:13.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:13.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:13.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:13.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:13.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:13.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:13.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:13.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:13.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:13.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:13.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:13.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:13.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:13.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:14.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:14.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:14.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:14.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:14.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:14.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:14.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:14.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:14.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:15.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597789 2021-07-26T06:45:15.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597789 '!=' null 2021-07-26T06:45:15.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:15.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:15.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:15.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:15.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:15.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:15.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:15.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:15.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:15.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:15.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:15.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:15.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:15.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:15.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:16.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:16.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:16.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:16.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:16.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:16.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:16.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:16.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:16.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:17.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21838731 2021-07-26T06:45:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21838731 '!=' null 2021-07-26T06:45:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:17.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:17.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:17.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:17.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:17.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:17.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:17.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:17.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:17.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:17.861 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:17.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:17.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:18.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:18.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:18.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:18.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:18.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:18.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:18.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:18.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:18.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:19.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091566 2021-07-26T06:45:19.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091566 '!=' null 2021-07-26T06:45:19.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:19.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:19.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:19.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:19.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:19.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:19.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:19.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:19.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:19.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:19.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:20.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:20.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:20.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:20.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:20.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:20.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:20.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:20.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:20.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:20.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:20.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:20.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:20.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:21.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091610 2021-07-26T06:45:21.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091610 '!=' null 2021-07-26T06:45:21.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:21.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:21.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:21.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:21.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:21.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:21.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:21.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:21.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:21.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:21.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:22.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:22.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:22.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:22.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:22.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:22.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:22.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:22.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:22.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:22.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:22.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:22.795 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:22.795 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:23.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344387 2021-07-26T06:45:23.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344387 '!=' null 2021-07-26T06:45:23.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:23.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:23.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:23.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:23.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:23.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:23.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:23.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:23.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:23.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:23.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:23.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:23.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:23.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:23.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:24.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:24.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:24.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:24.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:24.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:24.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:24.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:24.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:24.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:25.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14850246 2021-07-26T06:45:25.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14850246 '!=' null 2021-07-26T06:45:25.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:25.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:25.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:25.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:25.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:25.519 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:25.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:25.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:25.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:25.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:25.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:26.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:26.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:26.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:26.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:26.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:26.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:26.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:26.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:26.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:26.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:26.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:26.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:26.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:27.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344447 2021-07-26T06:45:27.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344447 '!=' null 2021-07-26T06:45:27.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:27.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:27.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:27.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:27.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:27.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:27.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:27.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:27.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:27.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:27.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:27.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:27.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:27.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:27.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:29.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:29.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:29.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:29.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:29.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:29.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:29.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:29.126 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:29.126 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:29.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597567 2021-07-26T06:45:29.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597567 '!=' null 2021-07-26T06:45:29.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:29.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:29.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:29.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:29.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:29.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:29.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:29.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:29.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:29.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:29.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:30.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:30.150 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:30.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:30.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:30.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:30.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:30.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:30.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:30.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:30.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:30.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:30.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:30.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:31.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344788 2021-07-26T06:45:31.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344788 '!=' null 2021-07-26T06:45:31.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:31.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:31.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:31.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:31.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:31.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:31.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:31.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:31.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:31.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:31.441 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:31.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:31.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:31.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:31.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:32.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:32.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:32.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:32.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:32.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:32.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:32.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:32.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:32.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:33.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344560 2021-07-26T06:45:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344560 '!=' null 2021-07-26T06:45:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:33.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:33.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:33.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:33.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:33.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:33.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:33.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:33.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:33.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:33.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:33.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:33.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:33.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:34.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:34.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:34.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:34.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:34.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:34.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:34.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:34.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:34.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:35.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15723991 2021-07-26T06:45:35.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15723991 '!=' null 2021-07-26T06:45:35.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:35.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:35.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:35.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:35.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:35.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:35.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:35.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:35.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:35.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:35.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:35.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:35.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:35.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:35.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:36.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:36.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:36.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:36.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:36.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:36.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:36.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:36.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:36.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:37.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091632 2021-07-26T06:45:37.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091632 '!=' null 2021-07-26T06:45:37.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:37.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:37.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:37.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:37.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:37.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:37.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:37.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:37.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:37.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:37.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:37.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:37.740 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:37.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:37.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:38.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:38.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:38.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:38.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:38.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:38.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:38.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:38.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:38.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:39.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091619 2021-07-26T06:45:39.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091619 '!=' null 2021-07-26T06:45:39.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:39.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:39.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:39.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:39.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:39.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:39.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:39.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:39.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:39.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:39.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:39.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:39.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:39.794 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:39.794 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:40.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:40.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:40.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:40.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:40.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:40.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:40.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:40.836 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:40.836 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:41.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965143 2021-07-26T06:45:41.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965143 '!=' null 2021-07-26T06:45:41.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:41.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:41.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:41.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:41.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:41.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:41.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:41.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:41.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:41.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:41.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:42.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:42.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:42.108 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:42.108 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:42.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:42.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:42.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:42.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:42.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:42.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:42.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:42.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:42.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:43.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217759 2021-07-26T06:45:43.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217759 '!=' null 2021-07-26T06:45:43.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:43.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:43.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:43.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:43.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:43.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:43.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:43.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:43.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:43.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:43.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:43.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:43.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:43.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:43.965 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:44.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:44.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:44.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:44.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:44.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:44.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:44.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:44.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:44.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:45.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15723571 2021-07-26T06:45:45.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15723571 '!=' null 2021-07-26T06:45:45.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:45.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:45.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:45.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:45.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:45.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:45.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:45.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:45.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:45.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:45.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:46.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:46.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:46.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:46.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:46.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:46.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:46.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:46.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:46.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:46.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:46.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:46.768 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:46.768 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:47.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217799 2021-07-26T06:45:47.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217799 '!=' null 2021-07-26T06:45:47.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:47.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:47.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:47.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:47.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:47.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:47.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:47.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:47.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:47.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:47.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:48.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:48.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:48.182 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:48.183 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:48.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:48.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:48.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:48.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:48.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:48.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:48.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:48.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:48.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:49.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344108 2021-07-26T06:45:49.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344108 '!=' null 2021-07-26T06:45:49.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:49.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:49.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:49.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:49.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:49.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:49.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:49.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:49.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:49.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:49.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:49.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:49.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:49.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:49.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:50.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:50.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:50.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:50.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:50.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:50.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:50.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:50.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:50.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:51.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344106 2021-07-26T06:45:51.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344106 '!=' null 2021-07-26T06:45:51.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:51.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:51.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:51.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:51.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:51.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:51.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:51.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:51.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:51.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:51.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:51.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:51.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:51.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:51.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:52.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:52.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:52.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:52.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:52.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:52.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:52.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:52.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:52.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:53.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470618 2021-07-26T06:45:53.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470618 '!=' null 2021-07-26T06:45:53.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:53.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:53.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:53.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:53.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:53.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:53.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:53.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:53.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:53.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:53.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:53.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:53.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:53.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:53.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:54.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:54.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:54.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:54.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:54.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:54.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:54.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:54.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:54.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:55.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976558 2021-07-26T06:45:55.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976558 '!=' null 2021-07-26T06:45:55.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:55.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:55.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:55.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:55.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:55.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:55.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:55.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:55.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:55.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:55.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:56.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:56.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:56.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:56.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:57.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:57.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:57.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:57.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:57.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:57.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:57.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:57.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:57.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:57.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470953 2021-07-26T06:45:57.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470953 '!=' null 2021-07-26T06:45:57.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:57.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:57.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:57.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:57.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:57.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:57.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:57.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:57.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:57.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:57.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:57.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:57.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:57.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:57.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:45:58.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:45:58.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:45:58.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:45:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:45:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:45:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:45:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:45:58.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:45:58.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:45:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15723772 2021-07-26T06:45:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15723772 '!=' null 2021-07-26T06:45:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:45:59.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:45:59.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:45:59.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:45:59.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:45:59.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:45:59.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:45:59.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:45:59.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:45:59.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:45:59.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:45:59.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:45:59.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:45:59.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:45:59.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:00.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:00.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:00.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:00.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:00.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:00.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:00.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:00.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:00.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:01.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471198 2021-07-26T06:46:01.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471198 '!=' null 2021-07-26T06:46:01.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:01.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:01.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:01.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:01.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:01.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:01.597 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:01.597 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:01.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:01.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:01.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:02.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:02.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:02.757 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:02.757 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:03.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:03.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:03.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:03.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:03.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:03.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:03.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:03.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:03.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:03.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344911 2021-07-26T06:46:03.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344911 '!=' null 2021-07-26T06:46:03.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:03.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:03.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:04.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:04.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:04.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:04.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:04.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:04.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:04.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:04.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:04.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:04.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:04.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:04.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:05.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:05.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:05.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:05.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:05.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:05.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:05.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:05.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:05.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:05.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14850600 2021-07-26T06:46:05.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14850600 '!=' null 2021-07-26T06:46:05.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:05.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:05.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:06.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:06.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:06.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:06.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:06.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:06.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:06.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:06.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:06.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:06.519 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:06.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:06.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:07.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:07.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:07.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345060 2021-07-26T06:46:07.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345060 '!=' null 2021-07-26T06:46:07.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:07.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:07.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:07.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:07.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:07.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:07.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:07.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:07.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:07.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:07.982 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:08.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:08.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:08.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:08.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:08.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:08.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:08.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:08.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:08.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:08.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:08.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:08.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:08.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:09.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344825 2021-07-26T06:46:09.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344825 '!=' null 2021-07-26T06:46:09.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:09.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:09.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:09.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:09.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:09.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:09.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:09.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:09.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:09.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:09.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:10.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:10.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:10.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:10.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:10.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:10.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:10.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:10.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:10.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:10.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:10.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:10.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:10.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:11.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21839514 2021-07-26T06:46:11.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21839514 '!=' null 2021-07-26T06:46:11.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:11.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:11.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:11.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:11.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:11.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:11.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:11.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:11.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:11.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:11.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:12.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:12.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:12.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:12.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:12.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:12.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:12.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:12.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:12.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:12.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:12.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:12.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:12.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:13.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092534 2021-07-26T06:46:13.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092534 '!=' null 2021-07-26T06:46:13.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:13.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:13.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:13.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:13.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:13.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:13.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:13.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:13.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:13.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:13.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:14.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:14.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:14.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:14.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:15.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:15.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:15.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:15.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:15.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:15.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:15.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:15.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:15.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:15.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15724383 2021-07-26T06:46:15.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15724383 '!=' null 2021-07-26T06:46:15.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:15.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:15.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:16.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:16.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:16.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:16.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:16.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:16.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:16.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:16.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:16.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:16.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:16.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:16.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:17.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:17.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:17.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:17.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:17.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:17.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:17.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:17.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:17.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:18.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218783 2021-07-26T06:46:18.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218783 '!=' null 2021-07-26T06:46:18.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:18.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:18.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:18.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:18.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:18.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:18.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:18.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:18.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:18.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:18.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:18.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:18.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:18.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:18.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:19.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:19.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:19.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:19.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:19.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:19.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:19.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:19.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:19.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:19.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218820 2021-07-26T06:46:19.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218820 '!=' null 2021-07-26T06:46:19.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:19.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:19.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:19.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:19.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:19.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:19.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:19.803 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:19.803 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:19.803 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:19.804 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:20.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:20.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:20.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:20.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:20.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:20.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:20.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:20.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:20.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:20.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:20.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:20.930 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:20.930 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:21.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21839955 2021-07-26T06:46:21.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21839955 '!=' null 2021-07-26T06:46:21.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:21.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:21.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:21.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:21.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:21.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:21.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:21.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:21.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:21.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:21.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:21.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:21.943 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:21.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:21.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:22.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:22.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:22.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:22.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:22.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:22.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:22.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:22.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:22.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:23.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092078 2021-07-26T06:46:23.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092078 '!=' null 2021-07-26T06:46:23.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:23.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:23.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:23.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:23.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:23.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:23.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:23.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:23.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:23.942 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:23.942 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:24.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:24.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:24.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:24.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:25.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:25.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:25.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:25.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:25.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:25.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:25.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:25.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:25.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:25.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597462 2021-07-26T06:46:25.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597462 '!=' null 2021-07-26T06:46:25.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:25.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:25.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:25.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:25.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:25.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:25.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:25.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:25.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:25.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:25.639 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:25.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:25.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:25.999 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:25.999 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:27.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:27.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:27.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:27.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:27.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:27.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:27.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:27.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:27.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:27.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965163 2021-07-26T06:46:27.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965163 '!=' null 2021-07-26T06:46:27.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:27.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:27.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:28.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:28.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:28.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:28.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:28.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:28.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:28.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:28.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:28.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:28.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:28.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:28.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:29.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:29.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:29.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:29.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:29.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:29.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:29.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:29.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:29.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:29.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091809 2021-07-26T06:46:29.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091809 '!=' null 2021-07-26T06:46:29.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:29.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:29.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:29.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:29.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:29.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:29.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:29.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:29.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:29.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:29.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:29.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:29.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:29.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:29.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:30.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:30.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:30.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:30.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:30.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:30.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:30.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:30.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:30.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:31.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965047 2021-07-26T06:46:31.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965047 '!=' null 2021-07-26T06:46:31.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:31.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:31.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:31.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:31.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:31.163 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:31.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:31.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:31.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:31.165 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:31.165 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:31.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:31.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:31.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:31.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:32.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:32.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:32.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:32.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:32.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:32.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:32.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:32.190 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:32.190 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:32.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470522 2021-07-26T06:46:32.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470522 '!=' null 2021-07-26T06:46:32.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:32.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:32.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:32.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:32.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:32.950 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:32.950 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:32.950 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:32.950 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:32.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:32.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:33.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:33.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:33.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:33.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:34.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:34.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:34.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:34.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:34.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:34.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:34.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:34.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:34.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:34.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976920 2021-07-26T06:46:34.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976920 '!=' null 2021-07-26T06:46:34.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:34.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:34.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:34.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:34.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:34.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:34.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:34.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:34.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:34.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:34.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:35.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:35.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:35.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:35.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:35.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:35.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:35.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:35.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:35.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:35.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:35.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:35.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:35.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:36.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344611 2021-07-26T06:46:36.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344611 '!=' null 2021-07-26T06:46:36.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:36.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:36.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:36.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:36.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:36.742 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:36.742 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:36.743 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:36.743 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:36.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:36.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:37.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:37.095 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:37.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:37.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:37.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:37.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:37.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:37.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:37.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:37.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:37.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:37.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:37.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:38.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344611 2021-07-26T06:46:38.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344611 '!=' null 2021-07-26T06:46:38.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:38.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:38.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:38.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:38.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:38.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:38.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:38.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:38.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:38.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:38.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:38.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:38.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:38.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:38.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:39.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:39.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:39.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:39.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:39.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:39.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:39.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:39.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:39.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:39.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471115 2021-07-26T06:46:39.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471115 '!=' null 2021-07-26T06:46:39.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:39.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:39.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:40.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:40.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:40.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:40.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:40.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:40.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:40.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:40.042 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:40.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:40.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:40.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:40.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:40.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:40.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:40.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:40.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:40.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:40.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:40.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:40.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:40.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:41.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217892 2021-07-26T06:46:41.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217892 '!=' null 2021-07-26T06:46:41.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:41.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:41.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:41.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:41.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:41.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:41.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:41.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:41.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:41.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:41.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:42.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:42.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:42.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:42.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:42.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:42.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:42.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:42.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:43.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217934 2021-07-26T06:46:43.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217934 '!=' null 2021-07-26T06:46:43.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:43.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:43.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:43.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:43.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:43.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:43.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:43.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:43.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:43.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:43.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:43.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:43.875 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:43.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:43.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:44.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:44.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:44.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:44.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:44.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:44.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:44.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:44.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:44.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:45.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597673 2021-07-26T06:46:45.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597673 '!=' null 2021-07-26T06:46:45.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:45.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:45.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:45.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:45.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:45.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:45.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:45.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:45.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:45.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:45.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:45.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:45.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:45.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:45.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:46.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:46.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:46.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:46.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:46.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:46.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:46.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:46.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:46.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:46.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21838715 2021-07-26T06:46:46.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21838715 '!=' null 2021-07-26T06:46:46.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:46.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:46.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:47.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:47.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:47.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:47.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:47.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:47.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:47.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:47.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:47.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:47.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:47.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:47.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:48.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:48.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:48.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:48.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:48.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:48.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:48.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:48.095 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:48.095 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:48.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21838937 2021-07-26T06:46:48.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21838937 '!=' null 2021-07-26T06:46:48.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:48.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:48.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:48.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:48.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:48.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:48.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:48.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:48.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:48.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:48.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:49.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:49.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:49.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:49.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:49.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:49.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:49.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:49.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:49.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:49.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:49.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:49.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:49.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:50.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21838937 2021-07-26T06:46:50.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21838937 '!=' null 2021-07-26T06:46:50.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:50.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:50.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:50.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:50.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:50.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:50.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:50.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:50.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:50.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:50.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:50.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:50.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:50.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:50.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:51.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:51.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:51.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:51.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:51.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:51.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:51.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:51.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:51.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:52.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21839021 2021-07-26T06:46:52.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21839021 '!=' null 2021-07-26T06:46:52.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:52.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:52.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:52.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:52.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:52.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:52.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:52.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:52.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:52.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:52.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:53.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:53.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:53.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:53.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:54.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:54.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:54.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:54.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:54.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:54.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:54.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:54.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471175 2021-07-26T06:46:54.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471175 '!=' null 2021-07-26T06:46:54.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:54.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:54.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:54.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:54.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:54.957 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:54.957 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:54.957 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:54.958 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:54.958 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:54.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:55.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:55.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:55.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:55.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:55.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:55.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:55.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:55.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:55.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:55.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:55.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:55.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:55.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:56.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965080 2021-07-26T06:46:56.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965080 '!=' null 2021-07-26T06:46:56.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:56.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:56.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:56.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:56.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:56.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:56.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:56.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:56.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:56.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:56.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:57.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:57.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:57.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:57.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:57.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:57.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:57.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:57.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:57.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:57.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:57.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:57.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:46:57.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:58.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965080 2021-07-26T06:46:58.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965080 '!=' null 2021-07-26T06:46:58.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:46:58.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:46:58.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:46:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:46:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:46:58.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:46:58.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:46:58.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:46:58.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:46:58.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:46:58.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:46:58.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:46:58.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:46:58.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:46:58.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:46:59.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:46:59.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:46:59.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:46:59.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:46:59.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:46:59.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:46:59.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:46:59.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:46:59.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:00.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344546 2021-07-26T06:47:00.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344546 '!=' null 2021-07-26T06:47:00.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:00.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:00.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:00.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:00.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:00.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:00.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:00.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:00.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:00.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:00.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:01.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:01.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:01.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:01.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:01.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:01.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:01.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:01.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:01.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:01.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:01.698 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:01.698 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:02.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344278 2021-07-26T06:47:02.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344278 '!=' null 2021-07-26T06:47:02.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:02.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:02.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:02.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:02.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:02.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:02.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:02.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:02.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:02.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:02.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:03.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:03.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:03.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:03.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:04.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:04.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:04.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:04.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:04.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:04.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:04.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:04.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:04.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:04.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976739 2021-07-26T06:47:04.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976739 '!=' null 2021-07-26T06:47:04.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:04.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:04.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:04.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:04.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:04.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:04.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:04.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:04.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:04.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:04.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:05.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:05.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:05.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:05.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:05.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:05.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:05.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:05.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:05.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:05.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:05.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:05.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:05.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:06.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470990 2021-07-26T06:47:06.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470990 '!=' null 2021-07-26T06:47:06.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:06.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:06.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:06.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:06.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:06.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:06.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:06.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:06.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:06.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:06.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:07.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:07.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:07.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:07.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:07.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:07.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:07.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:07.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:07.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:07.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:07.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:07.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:07.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:08.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471153 2021-07-26T06:47:08.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471153 '!=' null 2021-07-26T06:47:08.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:08.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:08.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:08.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:08.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:08.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:08.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:08.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:08.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:08.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:08.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:08.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:08.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:08.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:08.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:09.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:09.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:09.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:09.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:09.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:09.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:09.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:09.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:09.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:10.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344243 2021-07-26T06:47:10.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344243 '!=' null 2021-07-26T06:47:10.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:10.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:10.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:10.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:10.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:10.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:10.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:10.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:10.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:10.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:10.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:10.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:10.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:10.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:10.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:11.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:11.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:11.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:11.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:11.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:11.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:11.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:11.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:11.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:12.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344243 2021-07-26T06:47:12.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344243 '!=' null 2021-07-26T06:47:12.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:12.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:12.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:12.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:12.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:12.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:12.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:12.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:12.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:12.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:12.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:12.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:12.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:12.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:12.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:13.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:13.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:13.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:13.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:13.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:13.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:13.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:13.577 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:13.577 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:14.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15723916 2021-07-26T06:47:14.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15723916 '!=' null 2021-07-26T06:47:14.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:14.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:14.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:14.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:14.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:14.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:14.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:14.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:14.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:14.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:14.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:14.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:14.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:14.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:14.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:15.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:15.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:15.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:15.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:15.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:15.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:15.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:15.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:15.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:16.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091792 2021-07-26T06:47:16.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091792 '!=' null 2021-07-26T06:47:16.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:16.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:16.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:16.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:16.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:16.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:16.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:16.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:16.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:16.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:16.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:16.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:16.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:16.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:16.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:17.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:17.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:17.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:18.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091792 2021-07-26T06:47:18.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091792 '!=' null 2021-07-26T06:47:18.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:18.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:18.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:18.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:18.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:18.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:18.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:18.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:18.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:18.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:18.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:19.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:19.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:19.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:19.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:20.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:20.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:20.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:20.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:20.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:20.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:20.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:20.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:20.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:21.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=21839100 2021-07-26T06:47:21.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 21839100 '!=' null 2021-07-26T06:47:21.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:21.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:21.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:21.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:21.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:21.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:21.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:21.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:21.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:21.269 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:21.269 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:21.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:21.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:21.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:21.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:22.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:22.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:22.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:22.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:22.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:22.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:22.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:22.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:22.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:22.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=24897324 2021-07-26T06:47:22.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 24897324 '!=' null 2021-07-26T06:47:22.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:22.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:22.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:22.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:22.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:22.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:22.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:22.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:22.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:22.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:22.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:23.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:23.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:23.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:23.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:23.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:23.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:23.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:23.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:23.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:23.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:23.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:23.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:23.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:24.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=24897324 2021-07-26T06:47:24.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 24897324 '!=' null 2021-07-26T06:47:24.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:24.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:24.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:24.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:24.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:24.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:24.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:24.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:24.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:24.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:24.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:24.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:47:24.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:24.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:24.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:24.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 2 2021-07-26T06:47:24.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:47:24.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:24.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:24.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:24.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:24.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:24.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:24.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:25.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344984 2021-07-26T06:47:25.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344984 '!=' null 2021-07-26T06:47:25.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:25.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:25.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:25.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:25.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:25.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:25.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:25.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:25.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:25.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:25.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:25.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:25.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:25.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:25.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:26.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:26.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 0 2021-07-26T06:47:26.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:47:26.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=1 2021-07-26T06:47:26.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:26.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:26.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:26.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:26.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:26.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:26.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:26.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:26.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:26.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:26.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:26.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:26.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:26.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:27.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:27.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:27.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:27.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:27.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:27.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:28.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17034661 2021-07-26T06:47:28.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17034661 '!=' null 2021-07-26T06:47:28.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:28.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:28.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:28.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:28.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:28.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:28.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:28.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:28.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:28.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:28.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:28.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:28.686 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:28.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:28.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:29.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:29.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:29.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:29.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:29.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:29.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:29.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:29.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:29.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:29.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17034661 2021-07-26T06:47:29.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17034661 '!=' null 2021-07-26T06:47:29.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:29.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:29.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:29.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:29.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:29.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:29.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:29.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:29.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:29.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:29.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:30.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:30.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:30.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:30.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:30.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:30.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:30.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:30.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:30.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:30.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:30.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:30.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:30.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:31.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13806112 2021-07-26T06:47:31.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13806112 '!=' null 2021-07-26T06:47:31.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:31.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:31.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:31.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:31.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:31.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:31.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:31.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:31.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:31.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:31.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:31.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:31.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:31.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:31.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:32.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:32.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:32.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:32.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:32.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:32.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:32.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:32.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:32.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:32.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19518434 2021-07-26T06:47:32.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19518434 '!=' null 2021-07-26T06:47:32.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:32.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:32.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:33.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:33.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:33.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:33.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:33.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:33.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:33.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:33.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:33.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:33.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:33.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:33.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:33.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:33.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:33.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:33.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:33.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:33.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:33.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:33.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:33.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:34.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13977069 2021-07-26T06:47:34.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13977069 '!=' null 2021-07-26T06:47:34.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:34.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:34.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:34.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:34.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:34.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:34.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:34.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:34.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:34.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:34.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:35.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:35.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:35.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:35.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:35.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:35.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:35.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:35.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:35.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:35.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13977069 2021-07-26T06:47:35.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13977069 '!=' null 2021-07-26T06:47:35.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:35.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:35.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:36.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:36.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:36.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:36.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:36.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:36.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:36.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:36.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:36.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:36.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:36.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:36.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:37.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:37.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:37.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:37.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:37.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:37.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:37.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:37.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:37.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:38.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976963 2021-07-26T06:47:38.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976963 '!=' null 2021-07-26T06:47:38.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:38.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:38.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:38.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:38.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:38.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:38.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:38.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:38.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:38.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:38.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:38.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:38.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:38.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:38.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:39.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:39.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:39.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:39.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:39.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:39.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:39.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:39.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:39.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:40.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356378 2021-07-26T06:47:40.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356378 '!=' null 2021-07-26T06:47:40.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:40.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:40.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:40.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:40.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:40.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:40.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:40.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:40.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:40.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:40.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:40.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:40.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:40.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:40.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:41.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:41.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:41.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:41.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:41.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:41.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:41.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:41.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:41.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:41.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356378 2021-07-26T06:47:41.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356378 '!=' null 2021-07-26T06:47:41.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:41.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:41.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:41.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:41.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:41.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:41.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:41.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:41.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:41.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:41.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:42.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:42.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:42.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:42.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:42.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:42.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:42.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:42.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:42.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:42.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:42.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:42.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:42.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:43.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471518 2021-07-26T06:47:43.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471518 '!=' null 2021-07-26T06:47:43.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:43.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:43.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:43.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:43.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:43.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:43.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:43.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:43.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:43.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:43.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:43.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:43.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:43.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:43.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:44.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:44.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:44.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:44.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:44.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:44.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:44.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:44.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:44.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:44.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356252 2021-07-26T06:47:44.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356252 '!=' null 2021-07-26T06:47:44.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:44.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:44.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:44.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:44.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:44.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:44.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:44.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:44.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:44.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:44.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:45.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:45.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:45.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:45.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:45.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:45.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:45.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:45.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:45.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:45.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:45.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:45.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:45.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:46.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356545 2021-07-26T06:47:46.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356545 '!=' null 2021-07-26T06:47:46.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:46.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:46.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:46.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:46.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:46.364 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:46.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:46.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:46.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:46.366 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:46.366 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:46.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:46.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:46.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:46.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:47.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:47.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:47.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:47.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:47.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:47.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:47.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:47.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:47.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:47.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356545 2021-07-26T06:47:47.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356545 '!=' null 2021-07-26T06:47:47.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:47.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:47.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:47.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:47.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:47.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:47.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:47.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:47.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:47.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:47.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:48.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:48.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:48.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:48.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:48.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:48.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:48.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:48.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:48.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:48.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:48.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:48.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:48.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:49.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17472068 2021-07-26T06:47:49.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17472068 '!=' null 2021-07-26T06:47:49.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:49.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:49.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:49.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:49.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:49.341 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:49.342 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:49.342 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:49.342 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:49.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:49.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:49.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:49.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:49.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:49.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:50.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:50.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:50.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:50.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:50.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:50.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:50.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:50.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:50.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:50.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12230284 2021-07-26T06:47:50.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12230284 '!=' null 2021-07-26T06:47:50.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:50.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:50.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:50.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:50.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:50.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:50.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:50.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:50.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:50.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:50.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:51.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:51.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:51.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:51.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:51.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:51.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:51.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:51.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:51.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:51.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:51.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:51.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:51.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:52.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14676315 2021-07-26T06:47:52.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14676315 '!=' null 2021-07-26T06:47:52.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:52.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:52.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:52.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:52.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:52.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:52.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:52.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:52.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:52.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:52.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:52.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:52.836 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:52.837 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:52.837 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:53.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:53.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:53.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:53.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:53.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:53.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:53.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:53.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:53.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:53.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14676315 2021-07-26T06:47:53.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14676315 '!=' null 2021-07-26T06:47:53.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:53.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:53.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:53.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:53.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:53.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:53.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:53.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:53.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:53.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:53.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:54.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:54.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:54.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:54.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:54.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:54.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:54.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:54.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:54.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:54.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:54.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:54.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:54.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:55.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345424 2021-07-26T06:47:55.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345424 '!=' null 2021-07-26T06:47:55.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:55.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:55.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:55.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:55.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:55.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:55.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:55.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:55.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:55.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:55.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:55.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:55.644 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:55.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:55.646 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:56.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:56.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:56.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:56.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:56.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:56.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:56.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:56.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:56.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:56.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=9172416 2021-07-26T06:47:56.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 9172416 '!=' null 2021-07-26T06:47:56.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:56.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:56.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:56.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:56.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:56.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:56.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:56.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:56.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:56.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:56.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:57.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:47:57.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:57.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:57.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:57.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 2 2021-07-26T06:47:57.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:47:57.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:47:57.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:47:57.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:47:57.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:47:57.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:47:57.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:47:57.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:47:58.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=9172416 2021-07-26T06:47:58.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 9172416 '!=' null 2021-07-26T06:47:58.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:47:58.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T06:47:58.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:47:58.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:47:58.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:47:58.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:47:58.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:47:58.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:47:58.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:47:58.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:47:58.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:47:58.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T06:47:58.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:47:58.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:47:58.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:47:58.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 2 2021-07-26T06:47:58.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:47:58.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:47:58.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:522: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2021-07-26T06:47:58.896 INFO:tasks.workunit.client.0.smithi094.stdout:TEST PASSED 2021-07-26T06:47:58.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:526: TEST_recovery_pool_priority: echo TEST PASSED 2021-07-26T06:47:58.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: delete_pool test1 2021-07-26T06:47:58.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2021-07-26T06:47:58.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-07-26T06:47:59.430 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test1' does not exist 2021-07-26T06:47:59.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:530: TEST_recovery_pool_priority: delete_pool test2 2021-07-26T06:47:59.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-07-26T06:47:59.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-07-26T06:48:00.146 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test2' does not exist 2021-07-26T06:48:00.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:531: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2021-07-26T06:48:00.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:00.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:00.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:00.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:00.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:00.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:00.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: return 0 2021-07-26T06:48:00.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2021-07-26T06:48:00.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-07-26T06:48:00.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:48:00.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-07-26T06:48:00.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:00.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:00.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:00.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:00.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:00.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:00.483 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:48:00.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:48:00.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:48:00.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:48:00.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:48:00.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:48:00.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:48:00.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:48:00.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:48:00.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:48:00.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:48:00.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:48:00.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-07-26T06:48:01.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:48:01.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:01.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:01.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175221 2021-07-26T06:48:01.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:48:01.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:48:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-07-26T06:48:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-07-26T06:48:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-07-26T06:48:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-07-26T06:48:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-07-26T06:48:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T06:48:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-07-26T06:48:01.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T06:48:01.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T06:48:01.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T06:48:01.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T06:48:01.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T06:48:01.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T06:48:01.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T06:48:01.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T06:48:01.182 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T06:48:01.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T06:48:01.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T06:48:01.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T06:48:01.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:48:01.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T06:48:01.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T06:48:01.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T06:48:01.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T06:48:01.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T06:48:01.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-07-26T06:48:01.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T06:48:01.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:01.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:01.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175221 2021-07-26T06:48:01.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T06:48:01.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T06:48:01.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-07-26T06:48:01.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T06:48:01.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:01.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:01.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.175221 2021-07-26T06:48:01.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T06:48:01.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T06:48:01.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T06:48:01.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-07-26T06:48:01.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_priority td/osd-recovery-prio 2021-07-26T06:48:01.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:44: TEST_recovery_priority: local dir=td/osd-recovery-prio 2021-07-26T06:48:01.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:45: TEST_recovery_priority: local pools=10 2021-07-26T06:48:01.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:46: TEST_recovery_priority: local OSDS=5 2021-07-26T06:48:01.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local max_tries=10 2021-07-26T06:48:01.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2021-07-26T06:48:01.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-07-26T06:48:01.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T06:48:01.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T06:48:01.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T06:48:01.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-07-26T06:48:01.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-07-26T06:48:01.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T06:48:01.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:01.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:01.595 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:01.595 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:01.595 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:01.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:01.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T06:48:01.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T06:48:01.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T06:48:01.892 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:48:01.892 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:48:01.892 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T06:48:01.893 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T06:48:01.893 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:48:01.893 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:48:01.894 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:48:01.895 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:48:01.895 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:01.895 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:01.896 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-mon.a.asok 2021-07-26T06:48:01.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:48:01.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.175221/ceph-mon.a.asok config get fsid 2021-07-26T06:48:01.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T06:48:01.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T06:48:01.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T06:48:01.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T06:48:01.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T06:48:01.988 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T06:48:01.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T06:48:01.989 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T06:48:01.989 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:48:01.989 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:01.990 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:01.990 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-mon.a.asok 2021-07-26T06:48:01.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T06:48:01.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.175221/ceph-mon.a.asok config get mon_host 2021-07-26T06:48:02.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2021-07-26T06:48:02.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2021-07-26T06:48:02.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T06:48:02.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T06:48:02.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T06:48:02.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2021-07-26T06:48:02.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T06:48:02.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T06:48:02.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:02.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:02.990 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:02.990 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:02.990 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:02.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:02.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:48:02.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T06:48:03.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:51: TEST_recovery_priority: export CEPH_ARGS 2021-07-26T06:48:03.025 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: expr 5 - 1 2021-07-26T06:48:03.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: seq 0 4 2021-07-26T06:48:03.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:48:03.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2021-07-26T06:48:03.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-07-26T06:48:03.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:48:03.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T06:48:03.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:48:03.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2021-07-26T06:48:03.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:48:03.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:03.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:03.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:03.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-07-26T06:48:03.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-07-26T06:48:03.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:03.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:03.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:48:03.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-07-26T06:48:03.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:48:03.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:03.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:03.036 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:03.036 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:03.037 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:03.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:03.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:03.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:03.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:48:03.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:48:03.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-07-26T06:48:03.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-07-26T06:48:03.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:03.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:03.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:03.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:48:03.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:48:03.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2021-07-26T06:48:03.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:48:03.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6d7d61aa-8518-4ac0-9b1d-f99c7aa8abaa 2021-07-26T06:48:03.041 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 6d7d61aa-8518-4ac0-9b1d-f99c7aa8abaa 2021-07-26T06:48:03.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6d7d61aa-8518-4ac0-9b1d-f99c7aa8abaa' 2021-07-26T06:48:03.042 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:48:03.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjWv5g6E58AhAAB6aQMRlKOBPTCVy0UQo7PQ== 2021-07-26T06:48:03.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjWv5g6E58AhAAB6aQMRlKOBPTCVy0UQo7PQ=="}' 2021-07-26T06:48:03.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6d7d61aa-8518-4ac0-9b1d-f99c7aa8abaa -i td/osd-recovery-prio/0/new.json 2021-07-26T06:48:03.396 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:48:03.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2021-07-26T06:48:03.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCjWv5g6E58AhAAB6aQMRlKOBPTCVy0UQo7PQ== --osd-uuid 6d7d61aa-8518-4ac0-9b1d-f99c7aa8abaa 2021-07-26T06:48:03.435 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:03.424+0000 7f14f8355f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:03.437 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:03.424+0000 7f14f8355f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:03.439 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:03.428+0000 7f14f8355f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:03.439 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:03.428+0000 7f14f8355f40 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-07-26T06:48:03.439 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:03.428+0000 7f14f8355f40 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-07-26T06:48:03.439 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:03.428+0000 7f14f8355f40 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-07-26T06:48:03.495 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:03.484+0000 7f14f8355f40 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-07-26T06:48:08.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-07-26T06:48:08.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:48:08.829 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T06:48:08.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T06:48:08.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:48:09.336 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T06:48:09.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T06:48:09.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:48:09.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:48:09.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:48:09.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:09.368 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:09.356+0000 7f915893bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:09.370 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:09.360+0000 7f915893bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:09.378 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:09.360+0000 7f915893bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T06:48:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:09.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T06:48:09.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:09.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:09.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:09.723 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:48:09.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:09.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:09.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:48:09.916 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:09.904+0000 7f915893bf40 -1 Falling back to public interface 2021-07-26T06:48:10.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:11.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:11.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:11.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:11.094 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:48:11.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:11.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:48:11.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:12.205 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:12.192+0000 7f915893bf40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T06:48:12.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:12.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:12.462 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:48:12.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:48:12.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:12.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:48:12.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:13.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:13.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:13.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:48:13.850 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:48:13.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:13.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T06:48:14.207 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/198659,v1:127.0.0.1:6803/198659] [v2:127.0.0.1:6804/198659,v1:127.0.0.1:6805/198659] exists,up 6d7d61aa-8518-4ac0-9b1d-f99c7aa8abaa 2021-07-26T06:48:14.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:14.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:14.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:14.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:48:14.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2021-07-26T06:48:14.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-07-26T06:48:14.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:48:14.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T06:48:14.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:48:14.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2021-07-26T06:48:14.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:48:14.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:14.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:14.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:14.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-07-26T06:48:14.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-07-26T06:48:14.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:14.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:14.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:48:14.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-07-26T06:48:14.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:48:14.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:14.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:14.216 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:14.216 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:14.216 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:14.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:14.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:14.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:14.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:48:14.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:48:14.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-07-26T06:48:14.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-07-26T06:48:14.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:14.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:14.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:14.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:48:14.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:48:14.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2021-07-26T06:48:14.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:48:14.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1792c0e-5951-489f-8351-6afff02512ba 2021-07-26T06:48:14.220 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 e1792c0e-5951-489f-8351-6afff02512ba 2021-07-26T06:48:14.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e1792c0e-5951-489f-8351-6afff02512ba' 2021-07-26T06:48:14.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:48:14.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuWv5gETlJDRAA1LliNbvP9ksIot1STsY36Q== 2021-07-26T06:48:14.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuWv5gETlJDRAA1LliNbvP9ksIot1STsY36Q=="}' 2021-07-26T06:48:14.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1792c0e-5951-489f-8351-6afff02512ba -i td/osd-recovery-prio/1/new.json 2021-07-26T06:48:14.696 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:48:14.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2021-07-26T06:48:14.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCuWv5gETlJDRAA1LliNbvP9ksIot1STsY36Q== --osd-uuid e1792c0e-5951-489f-8351-6afff02512ba 2021-07-26T06:48:14.741 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:14.728+0000 7febebe5ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:14.743 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:14.732+0000 7febebe5ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:14.746 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:14.732+0000 7febebe5ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:14.747 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:14.736+0000 7febebe5ff40 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-07-26T06:48:14.747 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:14.736+0000 7febebe5ff40 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-07-26T06:48:14.747 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:14.736+0000 7febebe5ff40 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-07-26T06:48:14.826 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:14.812+0000 7febebe5ff40 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-07-26T06:48:20.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-07-26T06:48:20.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:48:20.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T06:48:20.618 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T06:48:20.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:48:21.079 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T06:48:21.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T06:48:21.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:21.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:48:21.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:48:21.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:48:21.111 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:21.100+0000 7ff5f066df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:21.115 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:21.104+0000 7ff5f066df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:21.116 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:21.104+0000 7ff5f066df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:21.404 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:21.392+0000 7ff5f066df40 -1 Falling back to public interface 2021-07-26T06:48:21.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T06:48:21.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:21.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T06:48:21.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:21.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:21.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:21.482 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:48:21.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:21.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:21.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:48:21.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:22.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:22.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:22.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:22.863 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:48:22.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:22.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:48:23.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:23.782 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:23.772+0000 7ff5f066df40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T06:48:24.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:24.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:24.236 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:48:24.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:48:24.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:24.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T06:48:24.674 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/200065,v1:127.0.0.1:6811/200065] [v2:127.0.0.1:6812/200065,v1:127.0.0.1:6813/200065] exists,up e1792c0e-5951-489f-8351-6afff02512ba 2021-07-26T06:48:24.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:24.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:24.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:24.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:48:24.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2021-07-26T06:48:24.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-07-26T06:48:24.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:48:24.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T06:48:24.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:48:24.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2021-07-26T06:48:24.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:48:24.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:24.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:24.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:24.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2021-07-26T06:48:24.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2021-07-26T06:48:24.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:24.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:24.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:48:24.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-07-26T06:48:24.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:48:24.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:24.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:24.683 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:24.684 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:24.684 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:24.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:24.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:24.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:24.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:48:24.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:48:24.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-07-26T06:48:24.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-07-26T06:48:24.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:24.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:24.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:24.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:48:24.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:48:24.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2021-07-26T06:48:24.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:48:24.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7addbda4-9ba0-42c3-83da-c759a4ed4803 2021-07-26T06:48:24.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7addbda4-9ba0-42c3-83da-c759a4ed4803' 2021-07-26T06:48:24.688 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 7addbda4-9ba0-42c3-83da-c759a4ed4803 2021-07-26T06:48:24.689 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:48:24.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4Wv5g1Gd8KRAA/9/im9xwOlu9mv1szQqmvQ== 2021-07-26T06:48:24.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4Wv5g1Gd8KRAA/9/im9xwOlu9mv1szQqmvQ=="}' 2021-07-26T06:48:24.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7addbda4-9ba0-42c3-83da-c759a4ed4803 -i td/osd-recovery-prio/2/new.json 2021-07-26T06:48:25.380 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:48:25.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2021-07-26T06:48:25.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC4Wv5g1Gd8KRAA/9/im9xwOlu9mv1szQqmvQ== --osd-uuid 7addbda4-9ba0-42c3-83da-c759a4ed4803 2021-07-26T06:48:25.433 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:25.424+0000 7fc151149f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:25.469 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:25.460+0000 7fc151149f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:25.471 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:25.464+0000 7fc151149f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:25.471 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:25.464+0000 7fc151149f40 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-07-26T06:48:25.471 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:25.464+0000 7fc151149f40 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-07-26T06:48:25.472 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:25.464+0000 7fc151149f40 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-07-26T06:48:25.577 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:25.568+0000 7fc151149f40 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2021-07-26T06:48:31.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2021-07-26T06:48:31.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:48:31.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T06:48:31.386 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T06:48:31.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:48:31.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T06:48:31.872 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T06:48:31.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:31.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:48:31.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:48:31.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:48:31.900 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:31.891+0000 7fc38d4d4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:31.903 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:31.895+0000 7fc38d4d4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:31.905 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:31.895+0000 7fc38d4d4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:32.200 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:32.191+0000 7fc38d4d4f40 -1 Falling back to public interface 2021-07-26T06:48:32.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T06:48:32.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:32.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T06:48:32.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:32.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:32.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:32.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:32.274 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:48:32.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:48:32.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:32.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:33.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:33.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:33.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:33.640 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:48:33.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:33.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:48:33.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:34.799 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:34.791+0000 7fc38d4d4f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T06:48:34.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:34.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:34.990 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:48:34.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:48:34.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:34.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:48:35.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:36.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:36.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:36.483 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:48:36.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:48:36.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:36.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T06:48:36.854 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/201445,v1:127.0.0.1:6819/201445] [v2:127.0.0.1:6820/201445,v1:127.0.0.1:6821/201445] exists,up 7addbda4-9ba0-42c3-83da-c759a4ed4803 2021-07-26T06:48:36.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:48:36.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2021-07-26T06:48:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-07-26T06:48:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:48:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T06:48:36.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:48:36.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2021-07-26T06:48:36.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:48:36.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:36.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:36.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:36.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2021-07-26T06:48:36.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2021-07-26T06:48:36.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:36.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:36.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:48:36.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-07-26T06:48:36.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:48:36.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:36.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:36.863 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:36.863 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:36.863 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:36.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:36.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:36.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:36.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:48:36.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:48:36.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-07-26T06:48:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-07-26T06:48:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:36.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:36.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:48:36.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:48:36.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2021-07-26T06:48:36.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:48:36.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e770077-fbbc-4f9f-aa57-3469661ed306 2021-07-26T06:48:36.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9e770077-fbbc-4f9f-aa57-3469661ed306' 2021-07-26T06:48:36.869 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 9e770077-fbbc-4f9f-aa57-3469661ed306 2021-07-26T06:48:36.870 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:48:36.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEWv5gwfdDNBAAtVSZx0x/Ace+JplNH9Vong== 2021-07-26T06:48:36.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEWv5gwfdDNBAAtVSZx0x/Ace+JplNH9Vong=="}' 2021-07-26T06:48:36.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e770077-fbbc-4f9f-aa57-3469661ed306 -i td/osd-recovery-prio/3/new.json 2021-07-26T06:48:37.352 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:48:37.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2021-07-26T06:48:37.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDEWv5gwfdDNBAAtVSZx0x/Ace+JplNH9Vong== --osd-uuid 9e770077-fbbc-4f9f-aa57-3469661ed306 2021-07-26T06:48:37.398 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:37.387+0000 7f88d5681f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:37.401 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:37.391+0000 7f88d5681f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:37.403 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:37.395+0000 7f88d5681f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:37.403 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:37.395+0000 7f88d5681f40 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-07-26T06:48:37.404 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:37.395+0000 7f88d5681f40 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-07-26T06:48:37.404 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:37.395+0000 7f88d5681f40 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-07-26T06:48:37.491 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:37.483+0000 7f88d5681f40 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2021-07-26T06:48:43.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2021-07-26T06:48:43.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:48:43.322 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T06:48:43.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T06:48:43.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:48:43.806 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T06:48:43.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T06:48:43.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:48:43.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:43.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:48:43.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:48:43.837 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:43.827+0000 7ff12b59ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:43.840 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:43.831+0000 7ff12b59ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:43.841 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:43.831+0000 7ff12b59ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:44.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T06:48:44.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:44.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T06:48:44.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:44.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:44.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:44.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:44.152 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:48:44.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:44.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:48:44.412 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:44.403+0000 7ff12b59ff40 -1 Falling back to public interface 2021-07-26T06:48:44.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:45.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:45.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:45.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:45.530 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:48:45.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:45.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:48:45.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:46.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:46.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:46.893 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:48:46.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:48:46.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:46.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:48:46.935 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:46.927+0000 7ff12b59ff40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T06:48:47.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:48.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:48.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:48.404 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:48:48.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:48:48.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:48.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T06:48:48.767 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/202851,v1:127.0.0.1:6827/202851] [v2:127.0.0.1:6828/202851,v1:127.0.0.1:6829/202851] exists,up 9e770077-fbbc-4f9f-aa57-3469661ed306 2021-07-26T06:48:48.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:48:48.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:48:48.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:48:48.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T06:48:48.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2021-07-26T06:48:48.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2021-07-26T06:48:48.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T06:48:48.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T06:48:48.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T06:48:48.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2021-07-26T06:48:48.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-07-26T06:48:48.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T06:48:48.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T06:48:48.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T06:48:48.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2021-07-26T06:48:48.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2021-07-26T06:48:48.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T06:48:48.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T06:48:48.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T06:48:48.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-07-26T06:48:48.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T06:48:48.777 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T06:48:48.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T06:48:48.778 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T06:48:48.778 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:48:48.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:48:48.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:48.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' 2021-07-26T06:48:48.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T06:48:48.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T06:48:48.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T06:48:48.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-07-26T06:48:48.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-07-26T06:48:48.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T06:48:48.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T06:48:48.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:48.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T06:48:48.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T06:48:48.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2021-07-26T06:48:48.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T06:48:48.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41273d28-4182-44ba-8199-281973fd7045 2021-07-26T06:48:48.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 41273d28-4182-44ba-8199-281973fd7045' 2021-07-26T06:48:48.788 INFO:tasks.workunit.client.0.smithi094.stdout:add osd4 41273d28-4182-44ba-8199-281973fd7045 2021-07-26T06:48:48.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T06:48:48.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQWv5gmKTaLhAAjO32ga/T3rEYgD6FRVTLyg== 2021-07-26T06:48:48.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQWv5gmKTaLhAAjO32ga/T3rEYgD6FRVTLyg=="}' 2021-07-26T06:48:48.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41273d28-4182-44ba-8199-281973fd7045 -i td/osd-recovery-prio/4/new.json 2021-07-26T06:48:49.419 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T06:48:49.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2021-07-26T06:48:49.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQWv5gmKTaLhAAjO32ga/T3rEYgD6FRVTLyg== --osd-uuid 41273d28-4182-44ba-8199-281973fd7045 2021-07-26T06:48:49.467 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:49.459+0000 7f8a2f99df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:49.471 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:49.463+0000 7f8a2f99df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:49.473 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:49.463+0000 7f8a2f99df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:49.474 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:49.463+0000 7f8a2f99df40 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-07-26T06:48:49.475 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:49.463+0000 7f8a2f99df40 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-07-26T06:48:49.475 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:49.463+0000 7f8a2f99df40 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-07-26T06:48:49.575 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:49.567+0000 7f8a2f99df40 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2021-07-26T06:48:55.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2021-07-26T06:48:55.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T06:48:55.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T06:48:55.418 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd4 key to auth repository 2021-07-26T06:48:55.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T06:48:55.899 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.4 2021-07-26T06:48:55.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T06:48:55.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=fd7c7925-61d8-4802-9e37-2cac8466c1ae --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.175221/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T06:48:55.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T06:48:55.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T06:48:55.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T06:48:55.929 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:55.919+0000 7ff97700df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:55.931 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:55.923+0000 7ff97700df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:55.932 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:55.923+0000 7ff97700df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T06:48:56.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T06:48:56.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T06:48:56.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T06:48:56.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T06:48:56.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T06:48:56.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:56.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T06:48:56.249 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T06:48:56.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:56.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:48:56.489 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:56.479+0000 7ff97700df40 -1 Falling back to public interface 2021-07-26T06:48:56.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:57.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:57.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:57.591 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T06:48:57.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T06:48:57.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:57.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:48:57.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:48:58.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:48:58.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:48:58.959 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T06:48:58.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T06:48:58.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:48:58.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:48:59.085 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T06:48:59.075+0000 7ff97700df40 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T06:48:59.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T06:49:00.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T06:49:00.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T06:49:00.403 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T06:49:00.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T06:49:00.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T06:49:00.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T06:49:00.776 INFO:tasks.workunit.client.0.smithi094.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/204257,v1:127.0.0.1:6835/204257] [v2:127.0.0.1:6836/204257,v1:127.0.0.1:6837/204257] exists,up 41273d28-4182-44ba-8199-281973fd7045 2021-07-26T06:49:00.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T06:49:00.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T06:49:00.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T06:49:00.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: seq 1 10 2021-07-26T06:49:00.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:00.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test1 1 1 2021-07-26T06:49:00.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-07-26T06:49:01.429 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test1' already exists 2021-07-26T06:49:01.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:02.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test1 size 2 2021-07-26T06:49:03.423 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 2 2021-07-26T06:49:03.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:03.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test2 1 1 2021-07-26T06:49:03.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-07-26T06:49:04.245 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test2' already exists 2021-07-26T06:49:04.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:05.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test2 size 2 2021-07-26T06:49:06.222 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 2 size to 2 2021-07-26T06:49:06.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:06.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test3 1 1 2021-07-26T06:49:06.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-07-26T06:49:06.891 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test3' already exists 2021-07-26T06:49:06.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:07.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-07-26T06:49:09.294 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 3 size to 2 2021-07-26T06:49:09.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:09.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test4 1 1 2021-07-26T06:49:09.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2021-07-26T06:49:10.065 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test4' already exists 2021-07-26T06:49:10.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:11.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test4 size 2 2021-07-26T06:49:12.305 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 4 size to 2 2021-07-26T06:49:12.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:12.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test5 1 1 2021-07-26T06:49:12.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2021-07-26T06:49:13.042 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test5' already exists 2021-07-26T06:49:13.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:14.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test5 size 2 2021-07-26T06:49:15.185 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 5 size to 2 2021-07-26T06:49:15.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:15.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test6 1 1 2021-07-26T06:49:15.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2021-07-26T06:49:15.858 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test6' already exists 2021-07-26T06:49:15.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:16.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test6 size 2 2021-07-26T06:49:18.110 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 6 size to 2 2021-07-26T06:49:18.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:18.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test7 1 1 2021-07-26T06:49:18.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2021-07-26T06:49:18.830 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test7' already exists 2021-07-26T06:49:18.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:19.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-07-26T06:49:20.917 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 7 size to 2 2021-07-26T06:49:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:20.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test8 1 1 2021-07-26T06:49:20.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2021-07-26T06:49:21.729 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test8' already exists 2021-07-26T06:49:21.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:22.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test8 size 2 2021-07-26T06:49:23.765 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 8 size to 2 2021-07-26T06:49:23.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:23.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test9 1 1 2021-07-26T06:49:23.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2021-07-26T06:49:24.522 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test9' already exists 2021-07-26T06:49:24.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:25.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test9 size 2 2021-07-26T06:49:26.656 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 9 size to 2 2021-07-26T06:49:26.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:26.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test10 1 1 2021-07-26T06:49:26.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2021-07-26T06:49:27.740 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test10' already exists 2021-07-26T06:49:27.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T06:49:28.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test10 size 2 2021-07-26T06:49:29.631 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 10 size to 2 2021-07-26T06:49:29.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: sleep 5 2021-07-26T06:49:34.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:65: TEST_recovery_priority: wait_for_clean 2021-07-26T06:49:34.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:49:34.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:34.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:34.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:34.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:34.649 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:34.650 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:34.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:34.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:34.651 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:34.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:34.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:34.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:34.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:34.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:35.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:35.254 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:49:35.254 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:49:35.255 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:49:35.255 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:49:35.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:35.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:35.256 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:35.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836499 2021-07-26T06:49:35.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836499 2021-07-26T06:49:35.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499' 2021-07-26T06:49:35.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:35.450 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:35.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672978 2021-07-26T06:49:35.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672978 2021-07-26T06:49:35.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2021-07-26T06:49:35.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:35.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:35.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509454 2021-07-26T06:49:35.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509454 2021-07-26T06:49:35.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509454' 2021-07-26T06:49:35.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:35.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:35.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345932 2021-07-26T06:49:35.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345932 2021-07-26T06:49:35.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509454 3-85899345932' 2021-07-26T06:49:35.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:35.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:36.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182410 2021-07-26T06:49:36.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182410 2021-07-26T06:49:36.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509454 3-85899345932 4-107374182410' 2021-07-26T06:49:36.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:36.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836499 2021-07-26T06:49:36.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:36.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:36.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836499 2021-07-26T06:49:36.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:36.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836499 2021-07-26T06:49:36.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-07-26T06:49:36.070 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836499 2021-07-26T06:49:36.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:36.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836497 -lt 21474836499 2021-07-26T06:49:36.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:49:37.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:49:37.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:37.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836499 -lt 21474836499 2021-07-26T06:49:37.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:37.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672978 2021-07-26T06:49:37.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:37.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:37.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672978 2021-07-26T06:49:37.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:37.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672978 2021-07-26T06:49:37.815 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672978 2021-07-26T06:49:37.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2021-07-26T06:49:37.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:38.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672978 -lt 42949672978 2021-07-26T06:49:38.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:38.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509454 2021-07-26T06:49:38.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:38.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:38.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509454 2021-07-26T06:49:38.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:38.204 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509454 2021-07-26T06:49:38.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509454 2021-07-26T06:49:38.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-07-26T06:49:38.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:38.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509455 -lt 64424509454 2021-07-26T06:49:38.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:38.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345932 2021-07-26T06:49:38.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:38.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:38.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345932 2021-07-26T06:49:38.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:38.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345932 2021-07-26T06:49:38.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2021-07-26T06:49:38.608 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345932 2021-07-26T06:49:38.608 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:38.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345932 -lt 85899345932 2021-07-26T06:49:38.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:38.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182410 2021-07-26T06:49:38.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:38.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:38.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182410 2021-07-26T06:49:38.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:38.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182410 2021-07-26T06:49:38.991 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182410 2021-07-26T06:49:38.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2021-07-26T06:49:38.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:39.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182410 -lt 107374182410 2021-07-26T06:49:39.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:39.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:39.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:39.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 10 == 0 2021-07-26T06:49:39.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:39.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:39.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:39.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:39.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:39.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:39.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:40.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=10 2021-07-26T06:49:40.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:40.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:40.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:40.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 10 = 10 2021-07-26T06:49:40.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:40.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:40.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:67: TEST_recovery_priority: ceph pg dump pgs 2021-07-26T06:49:41.120 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:49:41.121 INFO:tasks.workunit.client.0.smithi094.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:30.893607+0000 0'0 83:20 [2,1] 2 [2,1] 2 0'0 2021-07-26T06:49:27.256263+0000 0'0 2021-07-26T06:49:27.256263+0000 0 2021-07-26T06:49:41.121 INFO:tasks.workunit.client.0.smithi094.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:27.226933+0000 0'0 82:25 [1,2] 1 [1,2] 1 0'0 2021-07-26T06:49:24.256698+0000 0'0 2021-07-26T06:49:24.256698+0000 0 2021-07-26T06:49:41.121 INFO:tasks.workunit.client.0.smithi094.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:24.576834+0000 0'0 80:27 [3,4] 3 [3,4] 3 0'0 2021-07-26T06:49:21.466933+0000 0'0 2021-07-26T06:49:21.466933+0000 0 2021-07-26T06:49:41.121 INFO:tasks.workunit.client.0.smithi094.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:21.267156+0000 0'0 82:35 [1,4] 1 [1,4] 1 0'0 2021-07-26T06:49:18.538563+0000 0'0 2021-07-26T06:49:18.538563+0000 0 2021-07-26T06:49:41.122 INFO:tasks.workunit.client.0.smithi094.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:18.881278+0000 0'0 82:37 [4,2] 4 [4,2] 4 0'0 2021-07-26T06:49:15.574590+0000 0'0 2021-07-26T06:49:15.574590+0000 0 2021-07-26T06:49:41.122 INFO:tasks.workunit.client.0.smithi094.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:15.966789+0000 0'0 80:40 [3,4] 3 [3,4] 3 0'0 2021-07-26T06:49:12.779005+0000 0'0 2021-07-26T06:49:12.779005+0000 0 2021-07-26T06:49:41.122 INFO:tasks.workunit.client.0.smithi094.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:13.169363+0000 0'0 82:48 [4,0] 4 [4,0] 4 0'0 2021-07-26T06:49:09.723569+0000 0'0 2021-07-26T06:49:09.723569+0000 0 2021-07-26T06:49:41.122 INFO:tasks.workunit.client.0.smithi094.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:10.779908+0000 0'0 82:48 [1,2] 1 [1,2] 1 0'0 2021-07-26T06:49:06.707131+0000 0'0 2021-07-26T06:49:06.707131+0000 0 2021-07-26T06:49:41.123 INFO:tasks.workunit.client.0.smithi094.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:08.843169+0000 0'0 80:50 [3,1] 3 [3,1] 3 0'0 2021-07-26T06:49:03.973460+0000 0'0 2021-07-26T06:49:03.973460+0000 0 2021-07-26T06:49:41.123 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-07-26T06:49:04.378284+0000 0'0 82:61 [1,0] 1 [1,0] 1 0'0 2021-07-26T06:49:01.157992+0000 0'0 2021-07-26T06:49:01.157992+0000 0 2021-07-26T06:49:41.123 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:49:41.123 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:49:41.124 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:49:41.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:71: TEST_recovery_priority: local PG1 2021-07-26T06:49:41.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:72: TEST_recovery_priority: local POOLNUM1 2021-07-26T06:49:41.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:73: TEST_recovery_priority: local pool1 2021-07-26T06:49:41.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local chk_osd1_1 2021-07-26T06:49:41.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local chk_osd1_2 2021-07-26T06:49:41.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local PG2 2021-07-26T06:49:41.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local POOLNUM2 2021-07-26T06:49:41.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:79: TEST_recovery_priority: local pool2 2021-07-26T06:49:41.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local chk_osd2 2021-07-26T06:49:41.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local PG3 2021-07-26T06:49:41.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local POOLNUM3 2021-07-26T06:49:41.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:84: TEST_recovery_priority: local pool3 2021-07-26T06:49:41.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: seq 1 10 2021-07-26T06:49:41.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:41.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 1.0 --format=json 2021-07-26T06:49:41.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-07-26T06:49:41.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:49:41.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-07-26T06:49:41.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:49:41.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-07-26T06:49:41.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z '' ']' 2021-07-26T06:49:41.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: PG1=1.0 2021-07-26T06:49:41.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: POOLNUM1=1 2021-07-26T06:49:41.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:95: TEST_recovery_priority: pool1=test1 2021-07-26T06:49:41.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: chk_osd1_1=1 2021-07-26T06:49:41.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: chk_osd1_2=0 2021-07-26T06:49:41.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:41.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-07-26T06:49:41.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 2.0 --format=json 2021-07-26T06:49:41.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:49:41.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-07-26T06:49:41.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:49:41.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=1 2021-07-26T06:49:41.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-07-26T06:49:41.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2021-07-26T06:49:41.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2021-07-26T06:49:41.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:41.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-07-26T06:49:41.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 3.0 --format=json 2021-07-26T06:49:42.355 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:49:42.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-07-26T06:49:42.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:49:42.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-07-26T06:49:42.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-07-26T06:49:42.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2021-07-26T06:49:42.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: PG2=3.0 2021-07-26T06:49:42.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: POOLNUM2=3 2021-07-26T06:49:42.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:102: TEST_recovery_priority: pool2=test3 2021-07-26T06:49:42.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: chk_osd2=2 2021-07-26T06:49:42.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:42.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 4.0 --format=json 2021-07-26T06:49:42.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-07-26T06:49:42.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:49:42.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-07-26T06:49:42.755 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:49:42.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-07-26T06:49:42.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-07-26T06:49:42.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2021-07-26T06:49:42.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2021-07-26T06:49:42.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:42.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 5.0 --format=json 2021-07-26T06:49:42.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-07-26T06:49:43.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:49:43.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-07-26T06:49:43.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:49:43.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-07-26T06:49:43.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-07-26T06:49:43.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2021-07-26T06:49:43.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-07-26T06:49:43.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:43.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-07-26T06:49:43.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 6.0 --format=json 2021-07-26T06:49:43.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:49:43.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-07-26T06:49:43.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:49:43.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-07-26T06:49:43.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-07-26T06:49:43.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2021-07-26T06:49:43.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2021-07-26T06:49:43.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:43.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 7.0 --format=json 2021-07-26T06:49:43.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-07-26T06:49:43.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-07-26T06:49:43.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-07-26T06:49:43.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-07-26T06:49:43.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-07-26T06:49:43.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-07-26T06:49:43.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2021-07-26T06:49:43.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-07-26T06:49:43.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: PG3=7.0 2021-07-26T06:49:43.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: POOLNUM3=7 2021-07-26T06:49:43.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:108: TEST_recovery_priority: pool3=test7 2021-07-26T06:49:43.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: break 2021-07-26T06:49:43.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2021-07-26T06:49:43.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:114: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2021-07-26T06:49:43.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: seq 1 10 2021-07-26T06:49:43.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:43.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2021-07-26T06:49:43.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:43.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2021-07-26T06:49:43.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test2 2021-07-26T06:49:43.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-07-26T06:49:43.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-07-26T06:49:44.496 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test2' does not exist 2021-07-26T06:49:44.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:44.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2021-07-26T06:49:44.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:44.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2021-07-26T06:49:44.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test4 2021-07-26T06:49:44.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2021-07-26T06:49:44.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-07-26T06:49:45.125 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test4' does not exist 2021-07-26T06:49:45.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:45.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2021-07-26T06:49:45.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test5 2021-07-26T06:49:45.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2021-07-26T06:49:45.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-07-26T06:49:45.656 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test5' does not exist 2021-07-26T06:49:45.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:45.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2021-07-26T06:49:45.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test6 2021-07-26T06:49:45.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2021-07-26T06:49:45.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-07-26T06:49:46.266 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test6' does not exist 2021-07-26T06:49:46.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:46.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2021-07-26T06:49:46.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2021-07-26T06:49:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test8 2021-07-26T06:49:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2021-07-26T06:49:46.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-07-26T06:49:46.962 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test8' does not exist 2021-07-26T06:49:46.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:46.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2021-07-26T06:49:46.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test9 2021-07-26T06:49:46.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2021-07-26T06:49:46.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2021-07-26T06:49:47.671 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test9' does not exist 2021-07-26T06:49:47.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-07-26T06:49:47.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2021-07-26T06:49:47.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test10 2021-07-26T06:49:47.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2021-07-26T06:49:47.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2021-07-26T06:49:48.265 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test10' does not exist 2021-07-26T06:49:48.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:128: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2021-07-26T06:49:49.124 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 3 size to 1 2021-07-26T06:49:49.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:129: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2021-07-26T06:49:50.485 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 7 size to 1 2021-07-26T06:49:50.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:130: TEST_recovery_priority: wait_for_clean 2021-07-26T06:49:50.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T06:49:50.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:49:50.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:49:50.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:49:50.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:49:50.500 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:49:50.500 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:49:50.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:49:50.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:49:50.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:49:50.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:49:50.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:49:50.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:49:50.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:49:50.712 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:49:51.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:49:51.079 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:49:51.079 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:49:51.080 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:49:51.080 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:49:51.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:49:51.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:51.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:49:51.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836504 2021-07-26T06:49:51.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836504 2021-07-26T06:49:51.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504' 2021-07-26T06:49:51.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:51.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:49:51.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672983 2021-07-26T06:49:51.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672983 2021-07-26T06:49:51.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-42949672983' 2021-07-26T06:49:51.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:51.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:49:51.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509460 2021-07-26T06:49:51.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509460 2021-07-26T06:49:51.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-64424509460' 2021-07-26T06:49:51.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:51.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:49:51.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345937 2021-07-26T06:49:51.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345937 2021-07-26T06:49:51.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-64424509460 3-85899345937' 2021-07-26T06:49:51.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:49:51.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:49:51.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182415 2021-07-26T06:49:51.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182415 2021-07-26T06:49:51.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-64424509460 3-85899345937 4-107374182415' 2021-07-26T06:49:51.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:51.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836504 2021-07-26T06:49:51.797 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:51.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:49:51.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836504 2021-07-26T06:49:51.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:51.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836504 2021-07-26T06:49:51.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2021-07-26T06:49:51.801 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836504 2021-07-26T06:49:51.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:52.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836502 -lt 21474836504 2021-07-26T06:49:52.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:49:53.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:49:53.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:49:53.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836504 -lt 21474836504 2021-07-26T06:49:53.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:53.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672983 2021-07-26T06:49:53.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:53.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:49:53.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672983 2021-07-26T06:49:53.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:53.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672983 2021-07-26T06:49:53.527 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672983 2021-07-26T06:49:53.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2021-07-26T06:49:53.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:49:53.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672983 -lt 42949672983 2021-07-26T06:49:53.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:53.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509460 2021-07-26T06:49:53.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:53.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:49:53.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509460 2021-07-26T06:49:53.920 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:53.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509460 2021-07-26T06:49:53.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2021-07-26T06:49:53.922 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509460 2021-07-26T06:49:53.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:49:54.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509460 -lt 64424509460 2021-07-26T06:49:54.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:54.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345937 2021-07-26T06:49:54.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:54.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:49:54.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345937 2021-07-26T06:49:54.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:54.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345937 2021-07-26T06:49:54.309 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345937 2021-07-26T06:49:54.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2021-07-26T06:49:54.310 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:49:54.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345937 -lt 85899345937 2021-07-26T06:49:54.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:49:54.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182415 2021-07-26T06:49:54.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:49:54.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:49:54.680 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:49:54.680 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182415 2021-07-26T06:49:54.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182415 2021-07-26T06:49:54.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2021-07-26T06:49:54.682 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182415 2021-07-26T06:49:54.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:49:55.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182415 -lt 107374182415 2021-07-26T06:49:55.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:49:55.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:55.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:55.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 3 == 0 2021-07-26T06:49:55.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:49:55.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:49:55.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:49:55.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:49:55.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:49:55.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:49:55.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:49:55.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=3 2021-07-26T06:49:55.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:49:55.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:49:55.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:49:56.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 3 = 3 2021-07-26T06:49:56.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T06:49:56.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T06:49:56.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-07-26T06:49:56.409 INFO:tasks.workunit.client.0.smithi094.stderr:10+0 records in 2021-07-26T06:49:56.410 INFO:tasks.workunit.client.0.smithi094.stderr:10+0 records out 2021-07-26T06:49:56.410 INFO:tasks.workunit.client.0.smithi094.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0507323 s, 207 MB/s 2021-07-26T06:49:56.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: p=1 2021-07-26T06:49:56.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-07-26T06:49:56.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-07-26T06:49:56.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:49:56.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-07-26T06:49:57.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:49:57.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-07-26T06:49:57.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:49:57.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-07-26T06:49:58.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:49:58.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-07-26T06:49:59.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:49:59.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-07-26T06:49:59.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:49:59.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-07-26T06:50:01.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:01.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-07-26T06:50:01.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:01.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-07-26T06:50:02.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:02.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-07-26T06:50:03.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:03.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-07-26T06:50:04.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:04.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-07-26T06:50:04.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:04.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-07-26T06:50:05.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:05.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-07-26T06:50:06.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:06.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-07-26T06:50:07.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:07.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-07-26T06:50:08.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:08.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-07-26T06:50:08.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:08.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-07-26T06:50:10.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:10.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-07-26T06:50:10.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:10.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-07-26T06:50:11.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:11.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-07-26T06:50:12.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:12.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-07-26T06:50:13.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:13.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-07-26T06:50:13.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:13.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-07-26T06:50:14.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:14.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-07-26T06:50:16.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:16.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-07-26T06:50:17.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:17.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-07-26T06:50:17.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:17.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-07-26T06:50:18.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:18.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-07-26T06:50:19.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:19.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-07-26T06:50:20.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:20.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-07-26T06:50:20.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:20.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-07-26T06:50:22.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:22.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-07-26T06:50:22.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:22.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-07-26T06:50:23.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:23.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-07-26T06:50:23.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:23.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-07-26T06:50:24.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:24.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-07-26T06:50:25.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:25.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-07-26T06:50:26.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:26.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-07-26T06:50:27.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:27.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-07-26T06:50:28.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:28.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-07-26T06:50:28.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:28.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-07-26T06:50:29.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:29.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-07-26T06:50:30.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:30.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-07-26T06:50:31.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:31.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-07-26T06:50:32.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:32.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-07-26T06:50:32.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:32.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-07-26T06:50:33.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:33.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-07-26T06:50:35.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:35.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-07-26T06:50:36.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:36.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-07-26T06:50:37.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:37.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-07-26T06:50:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-07-26T06:50:38.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:38.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-07-26T06:50:39.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:39.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-07-26T06:50:40.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:40.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-07-26T06:50:41.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:41.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-07-26T06:50:41.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:41.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-07-26T06:50:42.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:42.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-07-26T06:50:43.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:43.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-07-26T06:50:44.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:44.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-07-26T06:50:44.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:44.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-07-26T06:50:45.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:45.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-07-26T06:50:46.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:46.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-07-26T06:50:47.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:47.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-07-26T06:50:48.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:48.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-07-26T06:50:48.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:48.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-07-26T06:50:50.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:50.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-07-26T06:50:50.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:50.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-07-26T06:50:51.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:51.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-07-26T06:50:52.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:52.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-07-26T06:50:53.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:53.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-07-26T06:50:53.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:53.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-07-26T06:50:55.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:55.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-07-26T06:50:56.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:56.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-07-26T06:50:57.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:57.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-07-26T06:50:57.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:57.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-07-26T06:50:58.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:58.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-07-26T06:50:59.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:50:59.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-07-26T06:51:00.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:00.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-07-26T06:51:01.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:01.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-07-26T06:51:02.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:02.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-07-26T06:51:02.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:02.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-07-26T06:51:03.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:03.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-07-26T06:51:04.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:04.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-07-26T06:51:05.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:05.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-07-26T06:51:05.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:05.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-07-26T06:51:06.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:06.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-07-26T06:51:07.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:07.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-07-26T06:51:08.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:08.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-07-26T06:51:08.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:08.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-07-26T06:51:09.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:09.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-07-26T06:51:10.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:10.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-07-26T06:51:11.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:11.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-07-26T06:51:12.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:12.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-07-26T06:51:12.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:12.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-07-26T06:51:14.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:14.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-07-26T06:51:16.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:16.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-07-26T06:51:17.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:17.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-07-26T06:51:18.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:18.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-07-26T06:51:19.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:19.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-07-26T06:51:20.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:20.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-07-26T06:51:20.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:20.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-07-26T06:51:21.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:21.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-07-26T06:51:22.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:22.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-07-26T06:51:24.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:24.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-07-26T06:51:25.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:25.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-07-26T06:51:26.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:26.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-07-26T06:51:26.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:26.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-07-26T06:51:27.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:27.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-07-26T06:51:28.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:28.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-07-26T06:51:29.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:29.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-07-26T06:51:30.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:30.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-07-26T06:51:31.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:31.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-07-26T06:51:31.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:31.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-07-26T06:51:32.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:32.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-07-26T06:51:33.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:33.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-07-26T06:51:34.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:34.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-07-26T06:51:34.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:34.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-07-26T06:51:36.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:36.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-07-26T06:51:37.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:37.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-07-26T06:51:38.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:38.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-07-26T06:51:38.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:38.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-07-26T06:51:40.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:40.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-07-26T06:51:40.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:40.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-07-26T06:51:41.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:41.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-07-26T06:51:42.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:42.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-07-26T06:51:43.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:43.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-07-26T06:51:43.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:43.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-07-26T06:51:44.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:44.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-07-26T06:51:45.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:45.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-07-26T06:51:46.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:46.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-07-26T06:51:47.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:47.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-07-26T06:51:48.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:48.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-07-26T06:51:48.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:48.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-07-26T06:51:49.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:49.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-07-26T06:51:50.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:50.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-07-26T06:51:51.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:51.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-07-26T06:51:52.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:52.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-07-26T06:51:52.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:52.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-07-26T06:51:53.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:53.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-07-26T06:51:54.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:54.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-07-26T06:51:55.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:55.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-07-26T06:51:56.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:56.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-07-26T06:51:58.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:58.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-07-26T06:51:58.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:58.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-07-26T06:51:59.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:51:59.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-07-26T06:52:00.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:00.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-07-26T06:52:01.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:01.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-07-26T06:52:01.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:01.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-07-26T06:52:02.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:02.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-07-26T06:52:03.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:03.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-07-26T06:52:04.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:04.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-07-26T06:52:05.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:05.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-07-26T06:52:06.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:06.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-07-26T06:52:06.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:06.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-07-26T06:52:07.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:07.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-07-26T06:52:08.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:08.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-07-26T06:52:09.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:09.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-07-26T06:52:10.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:10.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-07-26T06:52:11.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:11.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-07-26T06:52:11.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:11.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-07-26T06:52:12.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:12.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-07-26T06:52:13.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:13.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-07-26T06:52:14.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:14.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-07-26T06:52:15.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:15.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-07-26T06:52:15.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:15.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-07-26T06:52:17.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:17.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-07-26T06:52:18.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:18.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-07-26T06:52:19.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:19.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-07-26T06:52:20.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:20.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-07-26T06:52:21.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:21.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-07-26T06:52:21.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:21.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-07-26T06:52:22.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:22.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-07-26T06:52:23.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:23.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-07-26T06:52:24.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:24.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-07-26T06:52:24.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:24.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-07-26T06:52:25.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:25.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-07-26T06:52:26.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:26.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-07-26T06:52:27.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:27.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-07-26T06:52:28.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:28.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-07-26T06:52:28.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:28.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-07-26T06:52:29.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:29.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-07-26T06:52:30.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:30.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-07-26T06:52:31.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:31.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-07-26T06:52:32.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:32.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-07-26T06:52:32.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:32.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-07-26T06:52:33.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:33.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-07-26T06:52:34.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:34.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-07-26T06:52:35.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:35.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-07-26T06:52:36.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:36.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-07-26T06:52:39.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:39.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-07-26T06:52:39.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:39.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-07-26T06:52:40.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:40.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-07-26T06:52:41.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:41.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-07-26T06:52:42.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:42.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-07-26T06:52:43.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:43.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-07-26T06:52:43.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:43.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-07-26T06:52:44.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:44.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-07-26T06:52:45.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:45.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-07-26T06:52:46.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:46.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-07-26T06:52:47.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:47.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-07-26T06:52:47.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 1 + 1 2021-07-26T06:52:47.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=2 2021-07-26T06:52:47.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-07-26T06:52:47.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-07-26T06:52:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2021-07-26T06:52:48.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:48.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2021-07-26T06:52:48.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:48.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2021-07-26T06:52:49.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:49.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2021-07-26T06:52:50.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:50.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2021-07-26T06:52:50.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:50.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2021-07-26T06:52:51.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:51.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2021-07-26T06:52:52.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:52.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2021-07-26T06:52:52.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:52.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2021-07-26T06:52:53.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:53.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2021-07-26T06:52:53.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:53.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2021-07-26T06:52:53.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:53.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2021-07-26T06:52:54.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:54.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2021-07-26T06:52:55.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:55.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2021-07-26T06:52:56.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:56.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2021-07-26T06:52:56.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:56.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2021-07-26T06:52:56.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:56.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2021-07-26T06:52:57.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:57.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2021-07-26T06:52:57.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:57.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2021-07-26T06:52:57.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:57.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2021-07-26T06:52:58.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:58.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2021-07-26T06:52:59.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:59.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2021-07-26T06:52:59.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:52:59.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2021-07-26T06:53:00.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:00.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2021-07-26T06:53:00.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:00.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2021-07-26T06:53:00.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:00.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2021-07-26T06:53:01.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:01.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2021-07-26T06:53:02.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:02.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2021-07-26T06:53:02.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:02.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2021-07-26T06:53:03.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:03.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2021-07-26T06:53:03.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:03.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2021-07-26T06:53:03.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:03.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2021-07-26T06:53:04.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:04.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2021-07-26T06:53:05.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:05.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2021-07-26T06:53:05.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:05.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2021-07-26T06:53:06.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:06.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2021-07-26T06:53:06.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:06.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2021-07-26T06:53:07.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:07.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2021-07-26T06:53:08.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:08.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2021-07-26T06:53:08.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:08.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2021-07-26T06:53:08.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:08.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2021-07-26T06:53:09.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:09.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2021-07-26T06:53:09.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:09.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2021-07-26T06:53:10.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:10.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2021-07-26T06:53:10.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:10.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2021-07-26T06:53:11.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:11.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2021-07-26T06:53:11.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:11.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2021-07-26T06:53:12.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:12.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2021-07-26T06:53:12.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:12.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2021-07-26T06:53:13.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:13.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2021-07-26T06:53:13.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:13.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2021-07-26T06:53:14.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:14.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2021-07-26T06:53:14.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:14.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2021-07-26T06:53:15.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:15.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2021-07-26T06:53:15.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:15.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2021-07-26T06:53:16.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:16.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2021-07-26T06:53:16.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:16.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2021-07-26T06:53:17.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:17.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2021-07-26T06:53:17.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:17.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2021-07-26T06:53:18.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:18.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2021-07-26T06:53:18.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:18.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2021-07-26T06:53:19.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:19.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2021-07-26T06:53:20.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:20.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2021-07-26T06:53:20.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:20.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2021-07-26T06:53:20.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:20.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2021-07-26T06:53:21.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:21.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2021-07-26T06:53:21.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:21.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2021-07-26T06:53:22.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:22.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2021-07-26T06:53:22.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:22.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2021-07-26T06:53:23.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:23.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2021-07-26T06:53:23.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:23.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2021-07-26T06:53:24.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:24.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2021-07-26T06:53:24.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:24.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2021-07-26T06:53:25.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:25.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2021-07-26T06:53:25.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:25.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2021-07-26T06:53:26.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:26.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2021-07-26T06:53:26.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:26.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2021-07-26T06:53:27.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:27.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2021-07-26T06:53:27.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:27.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2021-07-26T06:53:27.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:27.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2021-07-26T06:53:28.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:28.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2021-07-26T06:53:28.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:28.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2021-07-26T06:53:29.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:29.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2021-07-26T06:53:30.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:30.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2021-07-26T06:53:32.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:32.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2021-07-26T06:53:32.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:32.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2021-07-26T06:53:33.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:33.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2021-07-26T06:53:33.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:33.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2021-07-26T06:53:34.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:34.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2021-07-26T06:53:34.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:34.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2021-07-26T06:53:34.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:34.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2021-07-26T06:53:35.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:35.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2021-07-26T06:53:36.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:36.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2021-07-26T06:53:36.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:36.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2021-07-26T06:53:36.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:36.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2021-07-26T06:53:37.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:37.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2021-07-26T06:53:38.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:38.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2021-07-26T06:53:38.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:38.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2021-07-26T06:53:39.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:39.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2021-07-26T06:53:39.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:39.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2021-07-26T06:53:39.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:39.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2021-07-26T06:53:40.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:40.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2021-07-26T06:53:40.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:40.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2021-07-26T06:53:41.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:41.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2021-07-26T06:53:42.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:42.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2021-07-26T06:53:42.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:42.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2021-07-26T06:53:43.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:43.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2021-07-26T06:53:43.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:43.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2021-07-26T06:53:44.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:44.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2021-07-26T06:53:45.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:45.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2021-07-26T06:53:45.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:45.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2021-07-26T06:53:46.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:46.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2021-07-26T06:53:46.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:46.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2021-07-26T06:53:47.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:47.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2021-07-26T06:53:48.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:48.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2021-07-26T06:53:48.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:48.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2021-07-26T06:53:48.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:48.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2021-07-26T06:53:49.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:49.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2021-07-26T06:53:49.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:49.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2021-07-26T06:53:50.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:50.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2021-07-26T06:53:50.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:50.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2021-07-26T06:53:51.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:51.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2021-07-26T06:53:51.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:51.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2021-07-26T06:53:52.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:52.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2021-07-26T06:53:52.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:52.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2021-07-26T06:53:52.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:52.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2021-07-26T06:53:54.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:54.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2021-07-26T06:53:54.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:54.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2021-07-26T06:53:54.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:54.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2021-07-26T06:53:55.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:55.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2021-07-26T06:53:56.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:56.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2021-07-26T06:53:56.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:56.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2021-07-26T06:53:57.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:57.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2021-07-26T06:53:57.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:57.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2021-07-26T06:53:58.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:58.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2021-07-26T06:53:58.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:58.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2021-07-26T06:53:58.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:58.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2021-07-26T06:53:59.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:53:59.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2021-07-26T06:54:00.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:00.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2021-07-26T06:54:00.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:00.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2021-07-26T06:54:00.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:00.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2021-07-26T06:54:01.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:01.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2021-07-26T06:54:01.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:01.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2021-07-26T06:54:02.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:02.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2021-07-26T06:54:03.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:03.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2021-07-26T06:54:03.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:03.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2021-07-26T06:54:03.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:03.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2021-07-26T06:54:04.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:04.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2021-07-26T06:54:04.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:04.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2021-07-26T06:54:05.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:05.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2021-07-26T06:54:06.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:06.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2021-07-26T06:54:06.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:06.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2021-07-26T06:54:06.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:06.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2021-07-26T06:54:07.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:07.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2021-07-26T06:54:08.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:08.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2021-07-26T06:54:08.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:08.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2021-07-26T06:54:09.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:09.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2021-07-26T06:54:09.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:09.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2021-07-26T06:54:10.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:10.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2021-07-26T06:54:10.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:10.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2021-07-26T06:54:10.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:10.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2021-07-26T06:54:12.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:12.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2021-07-26T06:54:12.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:12.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2021-07-26T06:54:12.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:12.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2021-07-26T06:54:13.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:13.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2021-07-26T06:54:13.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:13.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2021-07-26T06:54:13.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:13.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2021-07-26T06:54:14.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:14.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2021-07-26T06:54:15.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:15.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2021-07-26T06:54:15.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:15.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2021-07-26T06:54:16.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:16.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2021-07-26T06:54:16.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:16.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2021-07-26T06:54:16.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:16.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2021-07-26T06:54:17.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:17.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2021-07-26T06:54:18.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:18.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2021-07-26T06:54:18.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:18.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2021-07-26T06:54:19.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:19.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2021-07-26T06:54:20.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:20.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2021-07-26T06:54:21.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:21.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2021-07-26T06:54:21.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:21.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2021-07-26T06:54:22.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:22.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2021-07-26T06:54:22.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:22.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2021-07-26T06:54:22.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:22.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2021-07-26T06:54:23.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:23.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2021-07-26T06:54:23.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:23.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2021-07-26T06:54:24.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:24.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2021-07-26T06:54:24.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:24.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2021-07-26T06:54:25.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:25.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2021-07-26T06:54:25.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:25.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2021-07-26T06:54:26.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:26.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2021-07-26T06:54:26.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:26.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2021-07-26T06:54:27.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:27.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2021-07-26T06:54:27.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:27.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2021-07-26T06:54:28.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:28.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2021-07-26T06:54:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:28.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2021-07-26T06:54:28.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:28.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2021-07-26T06:54:29.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:29.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2021-07-26T06:54:30.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:30.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2021-07-26T06:54:30.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:30.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2021-07-26T06:54:31.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:31.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2021-07-26T06:54:31.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 2 + 1 2021-07-26T06:54:31.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=3 2021-07-26T06:54:31.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-07-26T06:54:31.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-07-26T06:54:31.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:31.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2021-07-26T06:54:32.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:32.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2021-07-26T06:54:32.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:32.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2021-07-26T06:54:33.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:33.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2021-07-26T06:54:33.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:33.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2021-07-26T06:54:34.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:34.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2021-07-26T06:54:34.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:34.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2021-07-26T06:54:35.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:35.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2021-07-26T06:54:35.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:35.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2021-07-26T06:54:36.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:36.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2021-07-26T06:54:37.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:37.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2021-07-26T06:54:38.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:38.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2021-07-26T06:54:38.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:38.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2021-07-26T06:54:39.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:39.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2021-07-26T06:54:39.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:39.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2021-07-26T06:54:40.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:40.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2021-07-26T06:54:40.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:40.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2021-07-26T06:54:41.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:41.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2021-07-26T06:54:41.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:41.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2021-07-26T06:54:42.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:42.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2021-07-26T06:54:42.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:42.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2021-07-26T06:54:43.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:43.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2021-07-26T06:54:43.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:43.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2021-07-26T06:54:44.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:44.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2021-07-26T06:54:44.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:44.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2021-07-26T06:54:45.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:45.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2021-07-26T06:54:46.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:46.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2021-07-26T06:54:46.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:46.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2021-07-26T06:54:46.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:46.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2021-07-26T06:54:47.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:47.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2021-07-26T06:54:47.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:47.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2021-07-26T06:54:48.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:48.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2021-07-26T06:54:49.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:49.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2021-07-26T06:54:49.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:49.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2021-07-26T06:54:50.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:50.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2021-07-26T06:54:50.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:50.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2021-07-26T06:54:50.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:50.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2021-07-26T06:54:51.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:51.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2021-07-26T06:54:52.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:52.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2021-07-26T06:54:52.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:52.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2021-07-26T06:54:53.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:53.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2021-07-26T06:54:53.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:53.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2021-07-26T06:54:53.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:53.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2021-07-26T06:54:54.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:54.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2021-07-26T06:54:55.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:55.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2021-07-26T06:54:55.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:55.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2021-07-26T06:54:55.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:55.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2021-07-26T06:54:56.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:56.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2021-07-26T06:54:57.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:57.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2021-07-26T06:54:58.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:58.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2021-07-26T06:54:58.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:58.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2021-07-26T06:54:58.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:58.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2021-07-26T06:54:59.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:59.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2021-07-26T06:54:59.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:54:59.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2021-07-26T06:55:00.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:00.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2021-07-26T06:55:00.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:00.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2021-07-26T06:55:01.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:01.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2021-07-26T06:55:01.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:01.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2021-07-26T06:55:02.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:02.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2021-07-26T06:55:02.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:02.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2021-07-26T06:55:02.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:02.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2021-07-26T06:55:03.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:03.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2021-07-26T06:55:04.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:04.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2021-07-26T06:55:04.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:04.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2021-07-26T06:55:04.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:04.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2021-07-26T06:55:05.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:05.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2021-07-26T06:55:05.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:05.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2021-07-26T06:55:06.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:06.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2021-07-26T06:55:07.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:07.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2021-07-26T06:55:07.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:07.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2021-07-26T06:55:08.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:08.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2021-07-26T06:55:09.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:09.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2021-07-26T06:55:10.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:10.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2021-07-26T06:55:10.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:10.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2021-07-26T06:55:10.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:10.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2021-07-26T06:55:11.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:11.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2021-07-26T06:55:11.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:11.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2021-07-26T06:55:12.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:12.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2021-07-26T06:55:13.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:13.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2021-07-26T06:55:13.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:13.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2021-07-26T06:55:13.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:13.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2021-07-26T06:55:14.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:14.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2021-07-26T06:55:14.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:14.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2021-07-26T06:55:15.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:15.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2021-07-26T06:55:16.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:16.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2021-07-26T06:55:16.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:16.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2021-07-26T06:55:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:16.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2021-07-26T06:55:17.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:17.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2021-07-26T06:55:17.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:17.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2021-07-26T06:55:18.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:18.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2021-07-26T06:55:18.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:18.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2021-07-26T06:55:19.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:19.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2021-07-26T06:55:19.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:19.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2021-07-26T06:55:20.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:20.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2021-07-26T06:55:21.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:21.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2021-07-26T06:55:21.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:21.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2021-07-26T06:55:22.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:22.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2021-07-26T06:55:22.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:22.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2021-07-26T06:55:23.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:23.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2021-07-26T06:55:23.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:23.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2021-07-26T06:55:23.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:23.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2021-07-26T06:55:24.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:24.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2021-07-26T06:55:25.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:25.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2021-07-26T06:55:25.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:25.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2021-07-26T06:55:26.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:26.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2021-07-26T06:55:26.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:26.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2021-07-26T06:55:26.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:26.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2021-07-26T06:55:27.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:27.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2021-07-26T06:55:28.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:28.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2021-07-26T06:55:28.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:28.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2021-07-26T06:55:28.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:28.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2021-07-26T06:55:29.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:29.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2021-07-26T06:55:30.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:30.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2021-07-26T06:55:31.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:31.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2021-07-26T06:55:31.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:31.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2021-07-26T06:55:32.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:32.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2021-07-26T06:55:32.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:32.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2021-07-26T06:55:33.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:33.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2021-07-26T06:55:34.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:34.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2021-07-26T06:55:34.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:34.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2021-07-26T06:55:35.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:35.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2021-07-26T06:55:35.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:35.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2021-07-26T06:55:36.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:36.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2021-07-26T06:55:36.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:36.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2021-07-26T06:55:37.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:37.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2021-07-26T06:55:37.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:37.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2021-07-26T06:55:38.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:38.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2021-07-26T06:55:38.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:38.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2021-07-26T06:55:38.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:38.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2021-07-26T06:55:39.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:39.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2021-07-26T06:55:39.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:39.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2021-07-26T06:55:40.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:40.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2021-07-26T06:55:40.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:40.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2021-07-26T06:55:41.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:41.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2021-07-26T06:55:41.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:41.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2021-07-26T06:55:42.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:42.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2021-07-26T06:55:42.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:42.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2021-07-26T06:55:42.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:42.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2021-07-26T06:55:43.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:43.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2021-07-26T06:55:44.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:44.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2021-07-26T06:55:44.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:44.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2021-07-26T06:55:45.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:45.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2021-07-26T06:55:46.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:46.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2021-07-26T06:55:46.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:46.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2021-07-26T06:55:47.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:47.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2021-07-26T06:55:47.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:47.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2021-07-26T06:55:47.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:47.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2021-07-26T06:55:48.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:48.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2021-07-26T06:55:48.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:48.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2021-07-26T06:55:49.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:49.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2021-07-26T06:55:50.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:50.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2021-07-26T06:55:50.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:50.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2021-07-26T06:55:50.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:50.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2021-07-26T06:55:51.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:51.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2021-07-26T06:55:51.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:51.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2021-07-26T06:55:52.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:52.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2021-07-26T06:55:52.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:52.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2021-07-26T06:55:53.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:53.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2021-07-26T06:55:53.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:53.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2021-07-26T06:55:54.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:54.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2021-07-26T06:55:54.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:54.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2021-07-26T06:55:54.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:54.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2021-07-26T06:55:55.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:55.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2021-07-26T06:55:56.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:56.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2021-07-26T06:55:56.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:56.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2021-07-26T06:55:57.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:57.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2021-07-26T06:55:58.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:58.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2021-07-26T06:55:59.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:59.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2021-07-26T06:55:59.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:59.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2021-07-26T06:55:59.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:55:59.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2021-07-26T06:56:00.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:00.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2021-07-26T06:56:00.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:00.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2021-07-26T06:56:01.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:01.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2021-07-26T06:56:02.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:02.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2021-07-26T06:56:02.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:02.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2021-07-26T06:56:02.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:02.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2021-07-26T06:56:03.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:03.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2021-07-26T06:56:03.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:03.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2021-07-26T06:56:04.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:04.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2021-07-26T06:56:04.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:04.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2021-07-26T06:56:05.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:05.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2021-07-26T06:56:05.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:05.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2021-07-26T06:56:06.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:06.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2021-07-26T06:56:06.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:06.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2021-07-26T06:56:06.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:06.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2021-07-26T06:56:08.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:08.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2021-07-26T06:56:08.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:08.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2021-07-26T06:56:08.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:08.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2021-07-26T06:56:09.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:09.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2021-07-26T06:56:10.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:10.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2021-07-26T06:56:10.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:10.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2021-07-26T06:56:11.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:11.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2021-07-26T06:56:11.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:11.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2021-07-26T06:56:12.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:12.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2021-07-26T06:56:12.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:12.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2021-07-26T06:56:13.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:13.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2021-07-26T06:56:14.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:14.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2021-07-26T06:56:14.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:14.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2021-07-26T06:56:14.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:14.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2021-07-26T06:56:15.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-07-26T06:56:15.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2021-07-26T06:56:15.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 3 + 1 2021-07-26T06:56:15.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=4 2021-07-26T06:56:15.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: get_not_primary test1 obj1-p1 2021-07-26T06:56:15.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test1 2021-07-26T06:56:15.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1-p1 2021-07-26T06:56:15.732 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test1 obj1-p1 2021-07-26T06:56:15.733 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test1 2021-07-26T06:56:15.733 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1-p1 2021-07-26T06:56:15.733 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test1 obj1-p1 2021-07-26T06:56:15.733 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T06:56:16.109 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T06:56:16.109 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-07-26T06:56:16.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T06:56:16.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: local otherosd=0 2021-07-26T06:56:16.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:145: TEST_recovery_priority: ceph pg dump pgs 2021-07-26T06:56:17.224 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:56:17.224 INFO:tasks.workunit.client.0.smithi094.stdout:7.0 199 0 0 0 0 2084569088 0 0 596 596 active+clean 2021-07-26T06:49:50.629052+0000 95'596 95:645 [1] 1 [1] 1 0'0 2021-07-26T06:49:18.538563+0000 0'0 2021-07-26T06:49:18.538563+0000 0 2021-07-26T06:56:17.224 INFO:tasks.workunit.client.0.smithi094.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T06:49:49.833221+0000 95'600 95:662 [1] 1 [1] 1 0'0 2021-07-26T06:49:06.707131+0000 0'0 2021-07-26T06:49:06.707131+0000 0 2021-07-26T06:56:17.225 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T06:49:04.378284+0000 95'600 95:668 [1,0] 1 [1,0] 1 0'0 2021-07-26T06:49:01.157992+0000 0'0 2021-07-26T06:49:01.157992+0000 0 2021-07-26T06:56:17.225 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:56:17.225 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:56:17.226 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:56:17.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: ERRORS=0 2021-07-26T06:56:17.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph osd set norecover 2021-07-26T06:56:18.021 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is set 2021-07-26T06:56:18.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ceph osd set noout 2021-07-26T06:56:18.792 INFO:tasks.workunit.client.0.smithi094.stderr:noout is set 2021-07-26T06:56:18.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:153: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-07-26T06:56:19.623 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 7 size to 2 2021-07-26T06:56:19.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:154: TEST_recovery_priority: sleep 2 2021-07-26T06:56:21.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:56:21.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:56:21.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:56:21.642 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:56:21.642 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:21.642 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:56:21.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:56:21.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:56:21.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:56:21.732 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:56:21.732 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:56:21.733 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:21.733 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:21.733 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:21.733 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:56:21.734 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:21.734 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:56:21.734 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:56:21.734 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:21.735 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:21.735 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:21.735 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:21.735 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:56:21.735 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:21.736 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:21.736 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:21.736 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:56:21.736 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:21.737 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:56:21.740 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: seq 1 10 2021-07-26T06:56:21.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-07-26T06:56:21.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-07-26T06:56:21.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-07-26T06:56:22.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 1 = 10 ']' 2021-07-26T06:56:22.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-07-26T06:56:24.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-07-26T06:56:24.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-07-26T06:56:24.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-07-26T06:56:24.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 2 = 10 ']' 2021-07-26T06:56:24.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-07-26T06:56:26.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-07-26T06:56:26.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-07-26T06:56:26.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-07-26T06:56:26.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 3 = 10 ']' 2021-07-26T06:56:26.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-07-26T06:56:28.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-07-26T06:56:28.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-07-26T06:56:28.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-07-26T06:56:29.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: break 2021-07-26T06:56:29.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:169: TEST_recovery_priority: flush_pg_stats 2021-07-26T06:56:29.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:29.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:29.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:29.540 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:56:29.540 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:56:29.540 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:56:29.541 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:56:29.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:29.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:29.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:29.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836586 2021-07-26T06:56:29.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836586 2021-07-26T06:56:29.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836586' 2021-07-26T06:56:29.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:29.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:29.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673064 2021-07-26T06:56:29.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673064 2021-07-26T06:56:29.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836586 1-42949673064' 2021-07-26T06:56:29.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:29.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:29.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509541 2021-07-26T06:56:29.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509541 2021-07-26T06:56:29.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836586 1-42949673064 2-64424509541' 2021-07-26T06:56:29.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:29.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:56:30.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899346019 2021-07-26T06:56:30.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899346019 2021-07-26T06:56:30.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836586 1-42949673064 2-64424509541 3-85899346019' 2021-07-26T06:56:30.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:30.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:56:30.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182496 2021-07-26T06:56:30.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182496 2021-07-26T06:56:30.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836586 1-42949673064 2-64424509541 3-85899346019 4-107374182496' 2021-07-26T06:56:30.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:30.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836586 2021-07-26T06:56:30.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:30.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:30.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836586 2021-07-26T06:56:30.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:30.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836586 2021-07-26T06:56:30.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2021-07-26T06:56:30.262 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836586 2021-07-26T06:56:30.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:30.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836584 -lt 21474836586 2021-07-26T06:56:30.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:31.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:56:31.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:31.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836586 -lt 21474836586 2021-07-26T06:56:31.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:31.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673064 2021-07-26T06:56:31.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:31.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:31.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673064 2021-07-26T06:56:31.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:31.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673064 2021-07-26T06:56:31.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2021-07-26T06:56:31.997 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949673064 2021-07-26T06:56:31.997 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:32.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673065 -lt 42949673064 2021-07-26T06:56:32.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:32.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509541 2021-07-26T06:56:32.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:32.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:32.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:32.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509541 2021-07-26T06:56:32.435 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509541 2021-07-26T06:56:32.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509541 2021-07-26T06:56:32.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2021-07-26T06:56:32.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:32.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509541 -lt 64424509541 2021-07-26T06:56:32.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:32.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899346019 2021-07-26T06:56:32.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:32.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:56:32.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899346019 2021-07-26T06:56:32.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:32.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899346019 2021-07-26T06:56:32.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899346019' 2021-07-26T06:56:32.823 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899346019 2021-07-26T06:56:32.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:56:33.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899346019 -lt 85899346019 2021-07-26T06:56:33.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:33.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182496 2021-07-26T06:56:33.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:33.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:56:33.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:33.217 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182496 2021-07-26T06:56:33.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182496 2021-07-26T06:56:33.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182496' 2021-07-26T06:56:33.219 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182496 2021-07-26T06:56:33.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:56:33.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182497 -lt 107374182496 2021-07-26T06:56:33.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:56:33.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:56:33.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:56:33.605 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:56:33.605 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:33.605 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:56:33.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:56:33.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:56:33.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:56:33.684 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:56:33.684 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:56:33.684 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:33.685 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:33.685 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:33.685 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:56:33.686 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:33.686 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:56:33.686 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:56:33.686 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:33.687 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:33.687 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:33.687 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:33.687 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:56:33.688 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:33.688 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:33.688 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:33.688 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:56:33.688 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:33.689 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:56:33.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: ceph osd out osd.0 2021-07-26T06:56:34.375 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T06:56:34.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: sleep 2 2021-07-26T06:56:36.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:174: TEST_recovery_priority: flush_pg_stats 2021-07-26T06:56:36.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:36.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:36.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:36.747 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:56:36.748 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:56:36.748 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:56:36.748 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:56:36.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:36.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:36.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:36.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836589 2021-07-26T06:56:36.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836589 2021-07-26T06:56:36.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836589' 2021-07-26T06:56:36.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:36.893 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:37.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673068 2021-07-26T06:56:37.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673068 2021-07-26T06:56:37.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836589 1-42949673068' 2021-07-26T06:56:37.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:37.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:37.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509545 2021-07-26T06:56:37.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509545 2021-07-26T06:56:37.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509545' 2021-07-26T06:56:37.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:37.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:56:37.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899346022 2021-07-26T06:56:37.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899346022 2021-07-26T06:56:37.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509545 3-85899346022' 2021-07-26T06:56:37.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:37.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:56:37.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182500 2021-07-26T06:56:37.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182500 2021-07-26T06:56:37.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509545 3-85899346022 4-107374182500' 2021-07-26T06:56:37.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:37.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836589 2021-07-26T06:56:37.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:37.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:37.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836589 2021-07-26T06:56:37.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:37.488 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836589 2021-07-26T06:56:37.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836589 2021-07-26T06:56:37.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2021-07-26T06:56:37.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:37.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836589 -lt 21474836589 2021-07-26T06:56:37.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:37.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:37.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673068 2021-07-26T06:56:37.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:37.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673068 2021-07-26T06:56:37.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:37.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673068 2021-07-26T06:56:37.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2021-07-26T06:56:37.855 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949673068 2021-07-26T06:56:37.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:38.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673068 -lt 42949673068 2021-07-26T06:56:38.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:38.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509545 2021-07-26T06:56:38.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:38.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:38.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509545 2021-07-26T06:56:38.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:38.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509545 2021-07-26T06:56:38.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509545' 2021-07-26T06:56:38.241 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509545 2021-07-26T06:56:38.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:38.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509543 -lt 64424509545 2021-07-26T06:56:38.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:39.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:56:39.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:39.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509545 -lt 64424509545 2021-07-26T06:56:39.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:39.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899346022 2021-07-26T06:56:39.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:39.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:56:39.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899346022 2021-07-26T06:56:39.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:39.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899346022 2021-07-26T06:56:39.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899346022' 2021-07-26T06:56:39.994 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899346022 2021-07-26T06:56:39.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:56:40.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899346023 -lt 85899346022 2021-07-26T06:56:40.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:40.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182500 2021-07-26T06:56:40.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:40.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:56:40.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182500 2021-07-26T06:56:40.404 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:40.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182500 2021-07-26T06:56:40.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182500' 2021-07-26T06:56:40.406 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182500 2021-07-26T06:56:40.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:56:40.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182500 -lt 107374182500 2021-07-26T06:56:40.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:56:40.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:56:40.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:56:40.800 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:56:40.800 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:40.800 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:56:40.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:56:40.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:56:40.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:56:40.876 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:56:40.876 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:56:40.876 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:40.877 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:40.877 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:56:40.877 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:40.877 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:56:40.878 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:56:40.878 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:40.878 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:56:40.878 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:56:40.878 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:40.879 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:40.879 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:40.879 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:40.879 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:56:40.880 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:56:40.880 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:40.880 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:56:40.880 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:56:40.880 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:40.881 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:40.881 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:40.881 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:40.881 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:56:40.881 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:40.882 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:40.882 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:40.882 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:56:40.882 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:40.883 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:56:40.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: ceph pg dump pgs 2021-07-26T06:56:41.236 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:56:41.236 INFO:tasks.workunit.client.0.smithi094.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-07-26T06:56:29.068153+0000 95'600 105:665 [1,4] 1 [1] 1 0'0 2021-07-26T06:49:18.538563+0000 0'0 2021-07-26T06:49:18.538563+0000 0 2021-07-26T06:56:41.237 INFO:tasks.workunit.client.0.smithi094.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T06:49:49.833221+0000 95'600 105:669 [1] 1 [1] 1 0'0 2021-07-26T06:49:06.707131+0000 0'0 2021-07-26T06:49:06.707131+0000 0 2021-07-26T06:56:41.237 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2021-07-26T06:56:36.384036+0000 95'600 106:682 [1,2] 1 [1] 1 0'0 2021-07-26T06:49:01.157992+0000 0'0 2021-07-26T06:49:01.157992+0000 0 2021-07-26T06:56:41.237 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:56:41.237 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:56:41.239 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:56:41.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-07-26T06:56:41.984 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 3 size to 2 2021-07-26T06:56:41.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: sleep 2 2021-07-26T06:56:43.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:180: TEST_recovery_priority: flush_pg_stats 2021-07-26T06:56:44.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:44.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:44.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:44.438 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:56:44.438 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:56:44.439 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:56:44.439 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:56:44.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:44.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:44.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:44.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836593 2021-07-26T06:56:44.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836593 2021-07-26T06:56:44.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836593' 2021-07-26T06:56:44.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:44.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:44.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673071 2021-07-26T06:56:44.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673071 2021-07-26T06:56:44.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836593 1-42949673071' 2021-07-26T06:56:44.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:44.814 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:44.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509548 2021-07-26T06:56:44.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509548 2021-07-26T06:56:44.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509548' 2021-07-26T06:56:44.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:44.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:56:45.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899346026 2021-07-26T06:56:45.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899346026 2021-07-26T06:56:45.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509548 3-85899346026' 2021-07-26T06:56:45.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:45.108 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:56:45.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182503 2021-07-26T06:56:45.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182503 2021-07-26T06:56:45.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509548 3-85899346026 4-107374182503' 2021-07-26T06:56:45.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:45.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836593 2021-07-26T06:56:45.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:45.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:45.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:45.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836593 2021-07-26T06:56:45.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836593 2021-07-26T06:56:45.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2021-07-26T06:56:45.275 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836593 2021-07-26T06:56:45.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:45.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836593 -lt 21474836593 2021-07-26T06:56:45.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:45.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673071 2021-07-26T06:56:45.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:45.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:45.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673071 2021-07-26T06:56:45.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:45.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673071 2021-07-26T06:56:45.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2021-07-26T06:56:45.637 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949673071 2021-07-26T06:56:45.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:46.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673071 -lt 42949673071 2021-07-26T06:56:46.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:46.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509548 2021-07-26T06:56:46.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:46.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:46.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509548 2021-07-26T06:56:46.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:46.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509548 2021-07-26T06:56:46.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2021-07-26T06:56:46.035 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509548 2021-07-26T06:56:46.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:46.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509548 -lt 64424509548 2021-07-26T06:56:46.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:46.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899346026 2021-07-26T06:56:46.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:46.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:56:46.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899346026 2021-07-26T06:56:46.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:46.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899346026 2021-07-26T06:56:46.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899346026' 2021-07-26T06:56:46.432 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899346026 2021-07-26T06:56:46.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:56:46.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899346024 -lt 85899346026 2021-07-26T06:56:46.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:47.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:56:47.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:56:48.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899346026 -lt 85899346026 2021-07-26T06:56:48.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:48.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182503 2021-07-26T06:56:48.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:48.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:56:48.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182503 2021-07-26T06:56:48.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:48.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182503 2021-07-26T06:56:48.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182503' 2021-07-26T06:56:48.300 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182503 2021-07-26T06:56:48.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:56:48.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182504 -lt 107374182503 2021-07-26T06:56:48.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:56:48.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:56:48.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:56:48.542 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:56:48.543 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:48.544 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:56:48.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:56:48.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:56:48.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:56:48.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:48.629 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:56:48.629 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:56:48.629 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:48.630 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:48.630 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:56:48.630 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:48.630 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:56:48.630 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:56:48.631 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:48.631 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:56:48.631 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:56:48.632 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:48.632 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:48.632 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:48.633 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:56:48.633 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:56:48.633 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:48.633 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:48.633 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:48.634 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:48.634 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:56:48.634 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:56:48.635 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:48.635 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:56:48.635 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:56:48.635 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:48.636 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:48.636 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:48.636 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:48.636 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:56:48.637 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:48.637 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:48.637 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:48.637 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:56:48.638 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:48.638 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:56:48.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: ceph pg dump pgs 2021-07-26T06:56:49.128 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T06:56:49.129 INFO:tasks.workunit.client.0.smithi094.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-07-26T06:56:29.068153+0000 95'600 109:668 [1,4] 1 [1] 1 0'0 2021-07-26T06:49:18.538563+0000 0'0 2021-07-26T06:49:18.538563+0000 0 2021-07-26T06:56:49.129 INFO:tasks.workunit.client.0.smithi094.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2021-07-26T06:56:43.423099+0000 95'600 110:679 [1,2] 1 [1] 1 0'0 2021-07-26T06:49:06.707131+0000 0'0 2021-07-26T06:49:06.707131+0000 0 2021-07-26T06:56:49.129 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2021-07-26T06:56:36.384036+0000 95'600 109:685 [1,2] 1 [1] 1 0'0 2021-07-26T06:49:01.157992+0000 0'0 2021-07-26T06:49:01.157992+0000 0 2021-07-26T06:56:49.129 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T06:56:49.130 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T06:56:49.131 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T06:56:49.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:49.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2021-07-26T06:56:49.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: PRIO=190 2021-07-26T06:56:49.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2021-07-26T06:56:49.183 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:49.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-07-26T06:56:49.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: eval 'ITEM="7.0"' 2021-07-26T06:56:49.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: ITEM=7.0 2021-07-26T06:56:49.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:194: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2021-07-26T06:56:49.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:49.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2021-07-26T06:56:49.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: PRIO=255 2021-07-26T06:56:49.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:200: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-07-26T06:56:49.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: seq 1 10 2021-07-26T06:56:49.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-07-26T06:56:49.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: ceph pg force-recovery 3.0 2021-07-26T06:56:49.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-07-26T06:56:49.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: break 2021-07-26T06:56:49.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:219: TEST_recovery_priority: sleep 2 2021-07-26T06:56:51.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:56:51.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:56:51.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:56:51.598 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:56:51.598 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:51.598 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:56:51.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:56:51.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:56:51.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:56:51.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:221: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:51.692 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:56:51.692 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:56:51.692 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:51.692 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:51.692 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:56:51.693 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:51.693 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:56:51.693 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:56:51.693 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:51.694 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:56:51.694 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:56:51.694 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:51.694 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:51.695 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:51.695 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:51.695 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:51.695 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 255, 2021-07-26T06:56:51.696 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:56:51.696 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:51.696 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:56:51.696 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:56:51.696 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:51.697 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:51.697 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:51.697 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:51.697 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:56:51.697 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:56:51.698 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:51.698 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:56:51.698 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:56:51.698 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:51.699 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:51.699 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:51.699 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:51.699 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:56:51.699 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:51.700 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:51.700 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:51.700 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:56:51.700 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:51.700 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:56:51.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:51.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-07-26T06:56:51.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: PRIO=255 2021-07-26T06:56:51.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-07-26T06:56:51.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:228: TEST_recovery_priority: flush_pg_stats 2021-07-26T06:56:51.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:56:51.740 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:56:52.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:56:52.090 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:56:52.090 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:56:52.091 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:56:52.091 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:56:52.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:56:52.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:52.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:56:52.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836596 2021-07-26T06:56:52.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836596 2021-07-26T06:56:52.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836596' 2021-07-26T06:56:52.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:52.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:56:52.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673075 2021-07-26T06:56:52.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673075 2021-07-26T06:56:52.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836596 1-42949673075' 2021-07-26T06:56:52.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:52.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:56:52.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509552 2021-07-26T06:56:52.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509552 2021-07-26T06:56:52.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836596 1-42949673075 2-64424509552' 2021-07-26T06:56:52.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:52.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:56:52.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899346029 2021-07-26T06:56:52.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899346029 2021-07-26T06:56:52.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836596 1-42949673075 2-64424509552 3-85899346029' 2021-07-26T06:56:52.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:56:52.661 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:56:52.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182507 2021-07-26T06:56:52.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182507 2021-07-26T06:56:52.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836596 1-42949673075 2-64424509552 3-85899346029 4-107374182507' 2021-07-26T06:56:52.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:52.798 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836596 2021-07-26T06:56:52.798 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:52.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:56:52.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836596 2021-07-26T06:56:52.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:52.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836596 2021-07-26T06:56:52.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2021-07-26T06:56:52.803 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836596 2021-07-26T06:56:52.804 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:53.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836594 -lt 21474836596 2021-07-26T06:56:53.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:56:54.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:56:54.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:56:54.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836596 -lt 21474836596 2021-07-26T06:56:54.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:54.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673075 2021-07-26T06:56:54.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:54.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:56:54.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673075 2021-07-26T06:56:54.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:54.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673075 2021-07-26T06:56:54.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2021-07-26T06:56:54.535 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949673075 2021-07-26T06:56:54.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:56:54.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673075 -lt 42949673075 2021-07-26T06:56:54.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:54.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509552 2021-07-26T06:56:54.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:54.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:56:54.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509552 2021-07-26T06:56:54.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:54.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509552 2021-07-26T06:56:54.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509552' 2021-07-26T06:56:54.915 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509552 2021-07-26T06:56:54.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:56:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509552 -lt 64424509552 2021-07-26T06:56:55.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:55.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899346029 2021-07-26T06:56:55.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:55.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:56:55.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899346029 2021-07-26T06:56:55.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:55.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899346029 2021-07-26T06:56:55.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899346029' 2021-07-26T06:56:55.282 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899346029 2021-07-26T06:56:55.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:56:55.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899346030 -lt 85899346029 2021-07-26T06:56:55.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:56:55.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182507 2021-07-26T06:56:55.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:56:55.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:56:55.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182507 2021-07-26T06:56:55.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:56:55.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182507 2021-07-26T06:56:55.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182507' 2021-07-26T06:56:55.649 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182507 2021-07-26T06:56:55.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:56:55.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182507 -lt 107374182507 2021-07-26T06:56:55.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:232: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2021-07-26T06:56:56.371 INFO:tasks.workunit.client.0.smithi094.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2021-07-26T06:56:56.372 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:56:56.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:233: TEST_recovery_priority: sleep 2 2021-07-26T06:56:58.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:56:58.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:56:58.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:56:58.389 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:56:58.389 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:56:58.390 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:56:58.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:56:58.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:56:58.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:56:58.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:58.489 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:56:58.490 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:56:58.491 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:58.491 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:58.491 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:56:58.492 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:58.492 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:56:58.493 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:56:58.493 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:58.493 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:56:58.494 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:56:58.494 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:58.495 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:58.495 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:58.496 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:56:58.496 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:56:58.497 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:58.497 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:58.497 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:58.498 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:58.498 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:56:58.499 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:56:58.499 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:56:58.500 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:56:58.501 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:56:58.501 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:56:58.501 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:58.502 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:56:58.502 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:56:58.503 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:56:58.503 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:56:58.504 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:56:58.504 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:56:58.504 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:56:58.505 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:56:58.505 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:56:58.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:58.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2021-07-26T06:56:58.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: PRIO=190 2021-07-26T06:56:58.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2021-07-26T06:56:58.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:58.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-07-26T06:56:58.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: eval 'ITEM="3.0"' 2021-07-26T06:56:58.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: ITEM=3.0 2021-07-26T06:56:58.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:245: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2021-07-26T06:56:58.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:56:58.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2021-07-26T06:56:58.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: PRIO=255 2021-07-26T06:56:58.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:251: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-07-26T06:56:58.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:258: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2021-07-26T06:56:58.966 INFO:tasks.workunit.client.0.smithi094.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2021-07-26T06:56:58.966 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:56:58.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:259: TEST_recovery_priority: sleep 5 2021-07-26T06:57:03.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:57:03.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:57:03.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:57:03.979 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:57:03.979 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:03.979 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:57:03.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:57:03.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:57:03.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:04.071 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:04.071 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:04.072 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:04.072 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:04.072 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:04.072 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:04.072 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:04.073 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:04.073 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:04.073 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:04.073 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:04.074 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:04.074 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:04.074 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:04.074 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:04.074 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:04.075 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:04.075 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:04.075 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:04.076 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:04.076 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:04.076 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:04.076 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:04.076 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:04.077 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:04.077 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:04.078 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:04.078 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:04.078 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:04.078 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:04.078 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:04.079 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:04.079 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:04.079 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:04.080 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:04.080 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:04.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: flush_pg_stats 2021-07-26T06:57:04.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:57:04.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:57:04.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:57:04.471 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:57:04.472 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:57:04.472 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:57:04.472 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:57:04.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:57:04.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:04.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:04.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836601 2021-07-26T06:57:04.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836601 2021-07-26T06:57:04.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836601' 2021-07-26T06:57:04.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:04.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:04.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673079 2021-07-26T06:57:04.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673079 2021-07-26T06:57:04.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836601 1-42949673079' 2021-07-26T06:57:04.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:04.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:04.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509556 2021-07-26T06:57:04.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509556 2021-07-26T06:57:04.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836601 1-42949673079 2-64424509556' 2021-07-26T06:57:04.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:04.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:57:05.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899346034 2021-07-26T06:57:05.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899346034 2021-07-26T06:57:05.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836601 1-42949673079 2-64424509556 3-85899346034' 2021-07-26T06:57:05.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:05.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:57:05.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182511 2021-07-26T06:57:05.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182511 2021-07-26T06:57:05.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836601 1-42949673079 2-64424509556 3-85899346034 4-107374182511' 2021-07-26T06:57:05.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:05.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836601 2021-07-26T06:57:05.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:05.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:05.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836601 2021-07-26T06:57:05.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:05.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836601 2021-07-26T06:57:05.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836601' 2021-07-26T06:57:05.232 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836601 2021-07-26T06:57:05.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:05.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836601 -lt 21474836601 2021-07-26T06:57:05.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:05.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673079 2021-07-26T06:57:05.593 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:05.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:57:05.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673079 2021-07-26T06:57:05.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:05.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673079 2021-07-26T06:57:05.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673079' 2021-07-26T06:57:05.600 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949673079 2021-07-26T06:57:05.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:57:05.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673079 -lt 42949673079 2021-07-26T06:57:05.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:05.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509556 2021-07-26T06:57:05.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:06.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:57:06.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509556 2021-07-26T06:57:06.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:06.003 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509556 2021-07-26T06:57:06.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509556 2021-07-26T06:57:06.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509556' 2021-07-26T06:57:06.004 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:57:06.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509556 -lt 64424509556 2021-07-26T06:57:06.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:06.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899346034 2021-07-26T06:57:06.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:06.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:57:06.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899346034 2021-07-26T06:57:06.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:06.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899346034 2021-07-26T06:57:06.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899346034' 2021-07-26T06:57:06.415 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899346034 2021-07-26T06:57:06.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:57:06.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899346034 -lt 85899346034 2021-07-26T06:57:06.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:06.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:06.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182511 2021-07-26T06:57:06.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:57:06.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:06.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182511 2021-07-26T06:57:06.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182511 2021-07-26T06:57:06.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182511' 2021-07-26T06:57:06.825 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182511 2021-07-26T06:57:06.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:57:07.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182509 -lt 107374182511 2021-07-26T06:57:07.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:08.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:57:08.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:57:08.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182512 -lt 107374182511 2021-07-26T06:57:08.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:264: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-07-26T06:57:08.909 INFO:tasks.workunit.client.0.smithi094.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2021-07-26T06:57:08.910 INFO:tasks.workunit.client.0.smithi094.stderr: 2021-07-26T06:57:08.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:265: TEST_recovery_priority: sleep 2 2021-07-26T06:57:10.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:57:10.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:57:10.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:57:10.926 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:57:10.927 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:10.927 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:57:10.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:57:10.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T06:57:10.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:11.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:57:11.028 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:11.028 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:11.029 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:11.029 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:11.029 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:11.029 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:11.029 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:11.030 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:11.030 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:11.030 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:11.030 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:11.031 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:11.031 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:11.031 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:11.031 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:11.031 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:11.032 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:11.032 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:11.032 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:11.032 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:11.032 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:11.033 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:11.033 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:11.033 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:11.033 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:11.034 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:11.034 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:11.035 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:11.035 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:11.035 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:11.035 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:11.035 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:11.036 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:11.036 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:11.036 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:11.036 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:11.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:57:11.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-07-26T06:57:11.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: PRIO=190 2021-07-26T06:57:11.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2021-07-26T06:57:11.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:57:11.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-07-26T06:57:11.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: eval 'ITEM="7.0"' 2021-07-26T06:57:11.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: ITEM=7.0 2021-07-26T06:57:11.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:277: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2021-07-26T06:57:11.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-07-26T06:57:11.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2021-07-26T06:57:11.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: PRIO=255 2021-07-26T06:57:11.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:283: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-07-26T06:57:11.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:290: TEST_recovery_priority: ceph osd unset noout 2021-07-26T06:57:11.840 INFO:tasks.workunit.client.0.smithi094.stderr:noout is unset 2021-07-26T06:57:11.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:291: TEST_recovery_priority: ceph osd unset norecover 2021-07-26T06:57:12.701 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is unset 2021-07-26T06:57:12.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T06:57:12.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T06:57:12.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T06:57:12.716 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T06:57:12.717 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T06:57:12.717 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T06:57:12.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T06:57:12.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations' 2021-07-26T06:57:12.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations' 2021-07-26T06:57:12.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T06:57:12.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T06:57:12.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T06:57:12.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T06:57:12.720 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T06:57:12.720 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T06:57:12.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T06:57:12.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T06:57:12.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T06:57:12.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T06:57:12.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T06:57:12.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T06:57:12.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T06:57:12.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T06:57:13.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T06:57:13.391 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T06:57:13.391 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T06:57:13.392 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T06:57:13.392 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T06:57:13.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T06:57:13.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:13.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T06:57:13.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836604 2021-07-26T06:57:13.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836604 2021-07-26T06:57:13.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836604' 2021-07-26T06:57:13.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:13.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T06:57:13.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949673083 2021-07-26T06:57:13.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949673083 2021-07-26T06:57:13.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836604 1-42949673083' 2021-07-26T06:57:13.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:13.689 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T06:57:13.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509560 2021-07-26T06:57:13.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509560 2021-07-26T06:57:13.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836604 1-42949673083 2-64424509560' 2021-07-26T06:57:13.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:13.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T06:57:13.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899346038 2021-07-26T06:57:13.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899346038 2021-07-26T06:57:13.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836604 1-42949673083 2-64424509560 3-85899346038' 2021-07-26T06:57:13.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T06:57:13.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T06:57:14.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182515 2021-07-26T06:57:14.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182515 2021-07-26T06:57:14.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836604 1-42949673083 2-64424509560 3-85899346038 4-107374182515' 2021-07-26T06:57:14.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:14.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:14.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836604 2021-07-26T06:57:14.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T06:57:14.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836604 2021-07-26T06:57:14.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:14.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836604 2021-07-26T06:57:14.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836604' 2021-07-26T06:57:14.133 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836604 2021-07-26T06:57:14.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:14.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836602 -lt 21474836604 2021-07-26T06:57:14.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T06:57:15.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T06:57:15.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T06:57:16.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836605 -lt 21474836604 2021-07-26T06:57:16.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:16.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949673083 2021-07-26T06:57:16.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:16.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T06:57:16.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949673083 2021-07-26T06:57:16.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:16.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949673083 2021-07-26T06:57:16.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949673083' 2021-07-26T06:57:16.102 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949673083 2021-07-26T06:57:16.102 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T06:57:16.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949673083 -lt 42949673083 2021-07-26T06:57:16.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:16.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509560 2021-07-26T06:57:16.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:16.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T06:57:16.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509560 2021-07-26T06:57:16.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:16.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509560 2021-07-26T06:57:16.577 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509560 2021-07-26T06:57:16.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509560' 2021-07-26T06:57:16.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T06:57:16.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509560 -lt 64424509560 2021-07-26T06:57:16.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:16.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899346038 2021-07-26T06:57:16.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:16.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T06:57:16.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899346038 2021-07-26T06:57:16.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:16.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899346038 2021-07-26T06:57:16.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899346038' 2021-07-26T06:57:16.969 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899346038 2021-07-26T06:57:16.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T06:57:17.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899346038 -lt 85899346038 2021-07-26T06:57:17.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T06:57:17.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182515 2021-07-26T06:57:17.449 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T06:57:17.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T06:57:17.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182515 2021-07-26T06:57:17.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T06:57:17.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182515 2021-07-26T06:57:17.454 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182515 2021-07-26T06:57:17.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182515' 2021-07-26T06:57:17.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T06:57:17.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182516 -lt 107374182515 2021-07-26T06:57:17.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T06:57:17.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:17.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:18.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 3 == 0 2021-07-26T06:57:18.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:18.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:18.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:18.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:18.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:18.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:18.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:18.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:18.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:18.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:18.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:19.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:19.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T06:57:19.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:57:19.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T06:57:19.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:19.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:19.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:19.563 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:19.564 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:19.564 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:19.564 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:19.564 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:19.565 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:19.565 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:19.565 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:19.566 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:19.566 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:19.566 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:19.566 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:19.566 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:19.567 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:19.567 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:19.567 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:19.567 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:19.567 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:19.568 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:19.568 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:19.568 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:19.568 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:19.568 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:19.569 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:19.569 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:19.569 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:19.569 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:19.569 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:19.570 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:19.570 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:19.570 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:19.570 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:19.571 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:19.571 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:19.571 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:19.571 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:19.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:19.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:19.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:19.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:19.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:19.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:19.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:19.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:19.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:20.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:20.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:20.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:20.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:20.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:20.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:20.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:20.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:20.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:20.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:20.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:20.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:20.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:21.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5785733 2021-07-26T06:57:21.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5785733 '!=' null 2021-07-26T06:57:21.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:21.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:21.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:21.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:21.108 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:21.109 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:21.109 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:21.109 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:21.109 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:21.110 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:21.110 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:21.110 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:21.110 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:21.110 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:21.111 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:21.111 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:21.111 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:21.111 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:21.112 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:21.112 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:21.112 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:21.112 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:21.112 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:21.113 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:21.113 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:21.113 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:21.113 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:21.113 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:21.114 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:21.114 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:21.114 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:21.114 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:21.114 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:21.115 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:21.115 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:21.115 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:21.115 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:21.116 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:21.116 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:21.116 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:21.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:21.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:21.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:21.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:21.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:21.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:21.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:21.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:21.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:21.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:21.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:21.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:21.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:22.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:22.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:22.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:22.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:22.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:22.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:22.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:22.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:22.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:22.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13299214 2021-07-26T06:57:22.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13299214 '!=' null 2021-07-26T06:57:22.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:22.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:22.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:22.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:22.800 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:22.800 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:22.801 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:22.801 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:22.801 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:22.801 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:22.801 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:22.802 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:22.802 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:22.802 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:22.802 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:22.803 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:22.803 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:22.803 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:22.803 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:22.803 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:22.804 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:22.804 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:22.804 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:22.804 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:22.805 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:22.805 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:22.805 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:22.805 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:22.805 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:22.806 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:22.806 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:22.806 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:22.807 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:22.807 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:22.807 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:22.807 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:22.808 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:22.808 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:22.808 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:22.808 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:22.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:22.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:22.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:22.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:22.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:22.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:22.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:22.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:22.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:23.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:23.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:23.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:23.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:23.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:23.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:23.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:23.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:23.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:23.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:23.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:23.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:23.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:24.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11101994 2021-07-26T06:57:24.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11101994 '!=' null 2021-07-26T06:57:24.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:24.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:24.346 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:24.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:24.435 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:24.436 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:24.436 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:24.436 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:24.437 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:24.437 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:24.438 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:24.438 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:24.438 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:24.439 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:24.439 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:24.440 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:24.440 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:24.440 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:24.441 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:24.441 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:24.446 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:24.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:24.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:24.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:24.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:24.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:24.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:24.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:24.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:24.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:25.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:25.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:25.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:25.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:25.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:25.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:25.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:25.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:25.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:25.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:25.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:25.686 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:25.686 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:26.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=9608892 2021-07-26T06:57:26.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 9608892 '!=' null 2021-07-26T06:57:26.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:26.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:26.273 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:26.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:26.358 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:26.358 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:26.359 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:26.359 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:26.359 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:26.359 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:26.359 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:26.360 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:26.360 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:26.360 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:26.360 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:26.361 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:26.361 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:26.361 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:26.362 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:26.362 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:26.362 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:26.362 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:26.362 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:26.363 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:26.363 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:26.363 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:26.363 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:26.364 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:26.364 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:26.365 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:26.365 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:26.365 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:26.366 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:26.366 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:26.366 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:26.366 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:26.367 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:26.367 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:26.367 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:26.367 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:26.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:26.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:26.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:26.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:26.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:26.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:26.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:26.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:26.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:26.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:26.829 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:26.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:26.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:27.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:27.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:27.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:27.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:27.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:27.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:27.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:27.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:27.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:28.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14849824 2021-07-26T06:57:28.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14849824 '!=' null 2021-07-26T06:57:28.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:28.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:28.109 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:28.109 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:28.194 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:28.194 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:28.195 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:28.195 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:28.195 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:28.195 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:28.196 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:28.196 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:28.196 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:28.196 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:28.197 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:28.197 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:28.197 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:28.197 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:28.197 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:28.198 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:28.198 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:28.198 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:28.198 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:28.199 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:28.199 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:28.199 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:28.200 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:28.200 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:28.200 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:28.200 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:28.201 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:28.201 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:28.201 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:28.202 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:28.202 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:28.202 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:28.202 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:28.202 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:28.203 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:28.203 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:28.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:28.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:28.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:28.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:28.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:28.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:28.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:28.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:28.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:28.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:28.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:28.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:28.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:29.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:29.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:29.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:29.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:29.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:29.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:29.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:29.126 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:29.126 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:29.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355677 2021-07-26T06:57:29.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355677 '!=' null 2021-07-26T06:57:29.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:29.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:29.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:29.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:29.719 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:29.719 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:29.719 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:29.720 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:29.720 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:29.720 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:29.720 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:29.720 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:29.721 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:29.721 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:29.721 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:29.721 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:29.721 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:29.722 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:29.722 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:29.722 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:29.722 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:29.724 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:29.724 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:29.724 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:29.724 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:29.725 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:29.725 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:29.725 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:29.725 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:29.725 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:29.726 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:29.726 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:29.726 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:29.726 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:29.726 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:29.727 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:29.727 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:29.727 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:29.727 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:29.727 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:29.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:29.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:29.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:29.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:29.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:29.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:29.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:29.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:29.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:30.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:30.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:30.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:30.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:30.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:30.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:30.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:30.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:30.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:30.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:30.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:30.854 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:30.854 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:31.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355677 2021-07-26T06:57:31.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355677 '!=' null 2021-07-26T06:57:31.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:31.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:31.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:31.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:31.459 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:31.459 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:31.459 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:31.460 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:31.460 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:31.460 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:31.460 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:31.461 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:31.461 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:31.461 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:31.461 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:31.461 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:31.462 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:31.462 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:31.462 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:31.463 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:31.463 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:31.463 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:31.463 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:31.464 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:31.464 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:31.464 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:31.464 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:31.465 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:31.465 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:31.465 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:31.465 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:31.466 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:31.466 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:31.467 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:31.467 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:31.467 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:31.468 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:31.468 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:31.468 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:31.468 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:31.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:31.570 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:31.570 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:31.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:31.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:31.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:31.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:31.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:31.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:31.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:31.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:32.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:32.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:32.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:32.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:32.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:32.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:32.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:32.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:32.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:33.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18343605 2021-07-26T06:57:33.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18343605 '!=' null 2021-07-26T06:57:33.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:33.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:33.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:33.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:33.091 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:33.092 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:33.092 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:33.092 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:33.092 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:33.093 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:33.093 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:33.093 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:33.094 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:33.095 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:33.096 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:33.097 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:33.097 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:33.097 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:33.098 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:33.098 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:33.098 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:33.098 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:33.099 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:33.099 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:33.099 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:33.099 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:33.100 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:33.100 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:33.100 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:33.100 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:33.100 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:33.101 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:33.101 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:33.101 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:33.101 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:33.102 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:33.102 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:33.102 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:33.102 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:33.102 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:33.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:33.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:33.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:33.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:33.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:33.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:33.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:33.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:33.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:33.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:33.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:33.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:33.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:34.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:34.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:34.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:34.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:34.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:34.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:34.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:34.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:34.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:34.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229064 2021-07-26T06:57:34.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229064 '!=' null 2021-07-26T06:57:34.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:34.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:34.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:34.559 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:34.639 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:34.640 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:34.640 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:34.640 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:34.640 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:34.641 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:34.641 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:34.641 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:34.641 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:34.641 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:34.642 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:34.642 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:34.642 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:34.642 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:34.642 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:34.643 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:34.643 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:34.643 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:34.643 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:34.644 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:34.644 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:34.644 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:34.644 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:34.644 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:34.645 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:34.645 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:34.645 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:34.645 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:34.645 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:34.646 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:34.646 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:34.646 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:34.646 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:34.647 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:34.647 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:34.647 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:34.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:34.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:34.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:34.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:34.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:34.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:34.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:34.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:34.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:35.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:35.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:35.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:35.125 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:35.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:35.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:35.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:35.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:35.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:35.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:35.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:35.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:35.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:36.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229155 2021-07-26T06:57:36.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229155 '!=' null 2021-07-26T06:57:36.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:36.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:36.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:36.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:36.318 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:36.319 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:36.319 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:36.319 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:36.319 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:36.320 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:36.320 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:36.320 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:36.320 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:36.321 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:36.321 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:36.321 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:36.321 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:36.322 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:36.322 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:36.322 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:36.322 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:36.323 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:36.323 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:36.323 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:36.323 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:36.323 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:36.324 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:36.324 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:36.324 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:36.324 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:36.325 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:36.325 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:36.325 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:36.325 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:36.326 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:36.326 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:36.326 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:36.326 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:36.327 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:36.327 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:36.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:36.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:36.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:36.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:36.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:36.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:36.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:36.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:36.431 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:36.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:36.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:36.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:36.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:37.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:37.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:37.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:37.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:37.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:37.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:37.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:37.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:37.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:37.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470063 2021-07-26T06:57:37.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470063 '!=' null 2021-07-26T06:57:37.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:37.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:37.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:37.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:37.951 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:37.951 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:37.951 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:37.952 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:37.952 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:37.952 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:37.952 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:37.952 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:37.953 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:37.953 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:37.953 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:37.954 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:37.954 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:37.954 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:37.954 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:37.955 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:37.955 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:37.955 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:37.955 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:37.955 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:37.956 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:37.956 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:37.956 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:37.956 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:37.956 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:37.957 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:37.957 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:37.957 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:37.957 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:37.957 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:37.958 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:37.958 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:37.958 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:37.958 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:37.958 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:37.959 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:37.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:38.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:38.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:38.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:38.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:38.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:38.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:38.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:38.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:38.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:38.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:38.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:38.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:38.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:38.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:38.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:38.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:38.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:39.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470063 2021-07-26T06:57:39.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470063 '!=' null 2021-07-26T06:57:39.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:39.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:39.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:39.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:39.548 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:39.549 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:39.549 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:39.549 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:39.550 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:39.550 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:39.550 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:39.550 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:39.551 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:39.551 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:39.551 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:39.551 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:39.552 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:39.552 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:39.552 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:39.552 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:39.552 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:39.553 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:39.553 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:39.553 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:39.553 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:39.554 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:39.554 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:39.554 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:39.554 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:39.554 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:39.555 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:39.555 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:39.556 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:39.557 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:39.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:39.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:39.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:39.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:39.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:39.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:39.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:39.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:39.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:39.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:39.998 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:39.999 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:39.999 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:40.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:40.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:40.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:40.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:40.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:40.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:40.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:40.591 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:40.591 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:41.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229200 2021-07-26T06:57:41.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229200 '!=' null 2021-07-26T06:57:41.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:41.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:41.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:41.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:41.118 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:41.118 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:41.118 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:41.118 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:41.119 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:41.119 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:41.119 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:41.119 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:41.120 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:41.120 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:41.120 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:41.120 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:41.120 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:41.121 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:41.121 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:41.121 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:41.121 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:41.122 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:41.122 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:41.122 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:41.122 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:41.122 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:41.123 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:41.123 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:41.123 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:41.123 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:41.124 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:41.124 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:41.124 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:41.124 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:41.124 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:41.125 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:41.125 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:41.125 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:41.125 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:41.126 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:41.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:41.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:41.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:41.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:41.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:41.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:41.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:41.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:41.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:41.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:41.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:41.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:41.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:42.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:42.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:42.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:42.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:42.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:42.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:42.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:42.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:42.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217540 2021-07-26T06:57:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217540 '!=' null 2021-07-26T06:57:42.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:42.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:42.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:42.640 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:42.721 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:42.722 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:42.722 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:42.722 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:42.722 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:42.723 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:42.723 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:42.723 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:42.723 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:42.724 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:42.724 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:42.724 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:42.724 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:42.724 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:42.725 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:42.725 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:42.725 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:42.725 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:42.726 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:42.726 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:42.726 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:42.726 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:42.726 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:42.727 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:42.727 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:42.727 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:42.727 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:42.728 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:42.728 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:42.728 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:42.728 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:42.728 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:42.729 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:42.729 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:42.729 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:42.729 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:42.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:42.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:42.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:42.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:42.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:42.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:42.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:42.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:42.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:43.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:43.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:43.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:43.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:43.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:43.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:43.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:43.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:43.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:43.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:43.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:43.835 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:43.836 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:44.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229301 2021-07-26T06:57:44.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229301 '!=' null 2021-07-26T06:57:44.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:44.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:44.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:44.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:44.440 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:44.498 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:44.498 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:44.498 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:44.499 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:44.499 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:44.499 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:44.499 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:44.499 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:44.500 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:44.500 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:44.500 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:44.500 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:44.500 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:44.501 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:44.501 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:44.501 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:44.501 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:44.501 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:44.502 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:44.502 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:44.502 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:44.502 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:44.503 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:44.503 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:44.503 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:44.503 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:44.503 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:44.504 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:44.504 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:44.504 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:44.504 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:44.504 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:44.505 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:44.505 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:44.505 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:44.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:44.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:44.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:44.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:44.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:44.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:44.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:44.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:44.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:44.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:44.902 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:44.902 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:44.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:45.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:45.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:45.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:45.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:45.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:45.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:45.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:45.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:45.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:45.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229279 2021-07-26T06:57:45.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229279 '!=' null 2021-07-26T06:57:45.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:45.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:45.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:45.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:45.929 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:45.929 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:45.930 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:45.930 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:45.930 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:45.931 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:45.931 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:45.931 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:45.931 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:45.932 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:45.932 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:45.932 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:45.933 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:45.933 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:45.933 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:45.934 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:45.934 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:45.935 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:45.935 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:45.936 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:45.936 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:45.936 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:45.937 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:45.937 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:45.938 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:45.938 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:45.939 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:45.939 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:45.939 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:45.940 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:45.940 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:45.941 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:45.941 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:45.942 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:45.942 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:45.942 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:45.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:46.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:46.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:46.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:46.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:46.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:46.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:46.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:46.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:46.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:46.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:46.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:46.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:46.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:46.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:46.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:46.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:46.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:46.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:46.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:46.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:46.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:47.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217516 2021-07-26T06:57:47.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217516 '!=' null 2021-07-26T06:57:47.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:47.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:47.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:47.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:47.627 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:47.627 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:47.628 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:47.628 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:47.628 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:47.629 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:47.629 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:47.629 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:47.629 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:47.629 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:47.630 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:47.630 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:47.630 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:47.630 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:47.631 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:47.631 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:47.631 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:47.632 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:47.632 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:47.632 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:47.632 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:47.633 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:47.633 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:47.634 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:47.634 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:47.634 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:47.634 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:47.635 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:47.635 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:47.635 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:47.635 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:47.635 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:47.636 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:47.636 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:47.636 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:47.636 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:47.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:47.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:47.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:47.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:47.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:47.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:47.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:47.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:47.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:48.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:48.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:48.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:48.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:48.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:48.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:48.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:48.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:48.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:48.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:48.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:48.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:48.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:49.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217516 2021-07-26T06:57:49.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217516 '!=' null 2021-07-26T06:57:49.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:49.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:49.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:49.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:49.355 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:49.356 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:49.357 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:49.357 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:49.357 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:49.358 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:49.358 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:49.359 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:49.360 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:49.361 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:49.362 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:49.362 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:49.363 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:49.364 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:49.364 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:49.365 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:49.366 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:49.366 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:49.367 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:49.368 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:49.368 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:49.369 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:49.369 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:49.369 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:49.369 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:49.370 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:49.370 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:49.370 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:49.370 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:49.371 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:49.371 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:49.371 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:49.371 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:49.372 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:49.372 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:49.372 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:49.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:49.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:49.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:49.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:49.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:49.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:49.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:49.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:49.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:49.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:49.870 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:49.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:49.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:50.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:50.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:50.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:50.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:50.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:50.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:50.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:50.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:50.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:50.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976539 2021-07-26T06:57:50.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976539 '!=' null 2021-07-26T06:57:50.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:50.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:50.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:50.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:51.037 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:51.038 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:51.038 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:51.038 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:51.038 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:51.039 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:51.039 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:51.039 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:51.039 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:51.039 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:51.040 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:51.040 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:51.040 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:51.041 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:51.041 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:51.041 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:51.042 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:51.042 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:51.042 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:51.042 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:51.043 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:51.043 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:51.043 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:51.043 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:51.044 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:51.044 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:51.044 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:51.045 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:51.045 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:51.045 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:51.045 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:51.045 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:51.046 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:51.046 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:51.046 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:51.046 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:51.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:51.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:51.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:51.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:51.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:51.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:51.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:51.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:51.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:51.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:51.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:51.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:51.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:52.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:52.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:52.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:52.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:52.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:52.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:52.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:52.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:52.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:52.957 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217764 2021-07-26T06:57:52.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217764 '!=' null 2021-07-26T06:57:52.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:52.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:52.958 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:52.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:53.045 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:53.045 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:53.045 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:53.046 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:53.046 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:53.046 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:53.046 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:53.047 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:53.047 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:53.047 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:53.047 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:53.047 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:53.048 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:53.048 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:53.048 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:53.048 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:53.048 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:53.049 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:53.049 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:53.049 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:53.049 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:53.050 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:53.050 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:53.050 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:53.050 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:53.050 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:53.051 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:53.051 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:53.051 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:53.051 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:53.051 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:53.052 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:53.052 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:53.052 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:53.052 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:53.053 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:53.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:53.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:53.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:53.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:53.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:53.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:53.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:53.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:53.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:53.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:53.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:53.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:53.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:54.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:54.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:54.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:54.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:54.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:54.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:54.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:54.064 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:54.064 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:54.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229470 2021-07-26T06:57:54.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229470 '!=' null 2021-07-26T06:57:54.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:54.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:54.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:54.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:54.632 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:54.633 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:54.633 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:54.633 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:54.633 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:54.634 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:54.634 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:54.634 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:54.634 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:54.634 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:54.635 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:54.635 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:54.635 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:54.635 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:54.635 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:54.636 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:54.636 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:54.636 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:54.636 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:54.637 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:54.637 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:54.637 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:54.637 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:54.637 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:54.638 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:54.638 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:54.638 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:54.638 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:54.638 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:54.639 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:54.639 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:54.639 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:54.639 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:54.639 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:54.640 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:54.640 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:54.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:54.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:54.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:54.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:54.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:54.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:54.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:54.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:54.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:55.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:55.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:55.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:55.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:55.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:55.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:55.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:55.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:55.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:55.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:55.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:55.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:55.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:56.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229597 2021-07-26T06:57:56.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229597 '!=' null 2021-07-26T06:57:56.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:56.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:56.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:56.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:56.186 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:56.187 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:56.187 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:56.187 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:56.187 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:56.188 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:56.188 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:56.188 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:56.188 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:56.189 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:56.189 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:56.189 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:56.189 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:56.189 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:56.190 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:56.190 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:56.190 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:56.190 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:56.191 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:56.191 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:56.191 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:56.191 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:56.191 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:56.192 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:56.192 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:56.192 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:56.192 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:56.193 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:56.193 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:56.193 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:56.193 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:56.193 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:56.194 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:56.194 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:56.194 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:56.194 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:56.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:56.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:56.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:56.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:56.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:56.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:56.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:56.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:56.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:56.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:56.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:56.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:56.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:57.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:57.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:57.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:57.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:57.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:57.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:57.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:57.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:57.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:57.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470900 2021-07-26T06:57:57.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470900 '!=' null 2021-07-26T06:57:57.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:57.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:57.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:57.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:57.734 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:57.734 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:57.735 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:57.735 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:57.736 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:57.736 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:57.736 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:57.737 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:57.737 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:57.738 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:57.738 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:57.739 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:57.739 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:57.740 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:57.740 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:57.740 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:57.741 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:57.741 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:57.741 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:57.742 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:57.742 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:57.742 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:57.742 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:57.743 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:57.743 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:57.743 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:57.744 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:57.744 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:57.744 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:57.745 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:57.745 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:57.745 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:57.745 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:57.745 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:57.746 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:57.746 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:57.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:57.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:57.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:57.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:57.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:57.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:57.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:57.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:57.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:58.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:58.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:58.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:58.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:57:58.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:57:58.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:57:58.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:57:58.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:57:58.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:57:58.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:57:58.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:57:58.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:57:58.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:57:59.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482549 2021-07-26T06:57:59.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482549 '!=' null 2021-07-26T06:57:59.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:57:59.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:59.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:57:59.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:57:59.419 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:57:59.419 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:57:59.419 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:59.419 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:59.420 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:57:59.420 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:59.420 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:57:59.420 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:57:59.421 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:59.421 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:57:59.421 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:59.421 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:59.422 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:59.422 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:59.422 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:57:59.422 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:57:59.423 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:59.423 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:59.423 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:59.423 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:59.423 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:57:59.424 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:57:59.424 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:57:59.424 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:57:59.424 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:57:59.424 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:57:59.425 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:59.425 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:57:59.425 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:57:59.425 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:57:59.426 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:57:59.426 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:57:59.426 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:57:59.426 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:57:59.426 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:57:59.427 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:57:59.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:57:59.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:57:59.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:57:59.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:57:59.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:57:59.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:57:59.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:57:59.531 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:57:59.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:57:59.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:57:59.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:57:59.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:57:59.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:00.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:00.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:00.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:00.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:00.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:00.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482549 2021-07-26T06:58:00.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482549 '!=' null 2021-07-26T06:58:00.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:00.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:00.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:00.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:01.000 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:01.001 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:01.001 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:01.002 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:01.002 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:01.003 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:01.004 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:01.004 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:01.005 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:01.005 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:01.006 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:01.006 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:01.007 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:01.007 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:01.008 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:01.008 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:01.008 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:01.009 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:01.009 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:01.010 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:01.010 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:01.011 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:01.011 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:01.012 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:01.012 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:01.012 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:01.013 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:01.014 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:01.014 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:01.014 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:01.015 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:01.015 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:01.015 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:01.015 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:01.015 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:01.016 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:01.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:01.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:01.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:01.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:01.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:01.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:01.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:01.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:01.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:01.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:01.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:01.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:01.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:02.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:02.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:02.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:02.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:02.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:02.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:02.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:02.196 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:02.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:02.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470998 2021-07-26T06:58:02.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470998 '!=' null 2021-07-26T06:58:02.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:02.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:02.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:02.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:02.714 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:02.715 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:02.715 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:02.715 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:02.716 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:02.716 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:02.716 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:02.716 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:02.716 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:02.717 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:02.717 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:02.717 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:02.717 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:02.718 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:02.718 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:02.718 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:02.718 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:02.719 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:02.719 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:02.719 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:02.719 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:02.719 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:02.720 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:02.720 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:02.720 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:02.720 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:02.720 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:02.721 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:02.721 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:02.721 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:02.721 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:02.722 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:02.722 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:02.722 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:02.722 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:02.722 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:02.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:02.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:02.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:02.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:02.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:02.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:02.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:02.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:02.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:03.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:03.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:03.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:03.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:03.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:03.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:03.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:03.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:03.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:03.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:03.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:03.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:03.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:04.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229756 2021-07-26T06:58:04.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229756 '!=' null 2021-07-26T06:58:04.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:04.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:04.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:04.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:04.123 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:04.123 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:04.124 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:04.124 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:04.124 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:04.124 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:04.125 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:04.125 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:04.125 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:04.125 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:04.125 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:04.126 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:04.126 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:04.126 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:04.126 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:04.127 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:04.127 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:04.127 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:04.127 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:04.127 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:04.128 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:04.128 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:04.128 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:04.128 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:04.128 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:04.129 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:04.129 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:04.129 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:04.129 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:04.129 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:04.130 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:04.130 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:04.130 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:04.130 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:04.130 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:04.131 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:04.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:04.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:04.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:04.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:04.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:04.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:04.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:04.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:04.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:04.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:04.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:04.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:04.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:05.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:05.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:05.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:05.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:05.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:05.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:05.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:05.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:05.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:06.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229731 2021-07-26T06:58:06.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229731 '!=' null 2021-07-26T06:58:06.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:06.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:06.203 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:06.203 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:06.276 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:06.276 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:06.276 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:06.276 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:06.277 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:06.277 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:06.277 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:06.277 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:06.278 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:06.278 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:06.278 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:06.278 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:06.279 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:06.279 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:06.279 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:06.279 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:06.279 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:06.280 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:06.280 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:06.280 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:06.280 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:06.280 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:06.281 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:06.281 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:06.281 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:06.281 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:06.281 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:06.282 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:06.282 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:06.282 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:06.282 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:06.283 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:06.283 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:06.283 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:06.283 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:06.284 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:06.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:06.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:06.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:06.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:06.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:06.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:06.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:06.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:06.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:06.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:06.757 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:06.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:06.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:07.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:07.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:07.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:08.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17468525 2021-07-26T06:58:08.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17468525 '!=' null 2021-07-26T06:58:08.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:08.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:08.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:08.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:08.194 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:08.194 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:08.194 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:08.195 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:08.195 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:08.195 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:08.195 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:08.196 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:08.196 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:08.196 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:08.196 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:08.196 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:08.197 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:08.197 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:08.197 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:08.197 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:08.197 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:08.198 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:08.198 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:08.198 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:08.198 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:08.198 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:08.199 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:08.199 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:08.199 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:08.199 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:08.199 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:08.200 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:08.200 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:08.200 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:08.200 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:08.200 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:08.201 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:08.201 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:08.201 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:08.201 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:08.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:08.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:08.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:08.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:08.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:08.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:08.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:08.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:08.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:08.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:08.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:08.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:08.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:09.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:09.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:09.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:09.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:09.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:09.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:09.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:09.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:09.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:09.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12227914 2021-07-26T06:58:09.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12227914 '!=' null 2021-07-26T06:58:09.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:09.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:09.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:09.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:09.757 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:09.757 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:09.758 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:09.758 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:09.759 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:09.759 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:09.759 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:09.759 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:09.762 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:09.762 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:09.762 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:09.763 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:09.763 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:09.764 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:09.764 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:09.764 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:09.764 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:09.765 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:09.765 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:09.765 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:09.766 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:09.766 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:09.767 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:09.767 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:09.767 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:09.768 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:09.768 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:09.768 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:09.768 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:09.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:09.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:09.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:09.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:09.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:09.870 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:09.870 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:09.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:09.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:10.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:10.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:10.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:10.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:11.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:11.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:11.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:11.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:11.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:11.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:11.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:11.091 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:11.091 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:11.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19215336 2021-07-26T06:58:11.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19215336 '!=' null 2021-07-26T06:58:11.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:11.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:11.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:11.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:11.734 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:11.735 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:11.735 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:11.735 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:11.735 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:11.736 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:11.736 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:11.736 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:11.736 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:11.737 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:11.737 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:11.738 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:11.738 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:11.738 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:11.738 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:11.738 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:11.739 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:11.739 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:11.739 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:11.739 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:11.740 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:11.740 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:11.740 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:11.740 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:11.741 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:11.741 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:11.741 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:11.741 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:11.742 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:11.742 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:11.742 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:11.742 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:11.743 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:11.743 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:11.744 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:11.744 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:11.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:11.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:11.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:11.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:11.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:11.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:11.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:11.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:11.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:12.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:12.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:12.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:12.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:12.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:12.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:12.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:12.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:12.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:12.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:12.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:12.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:12.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:13.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19215336 2021-07-26T06:58:13.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19215336 '!=' null 2021-07-26T06:58:13.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:13.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:13.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:13.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:13.662 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:13.663 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:13.663 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:13.663 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:13.663 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:13.664 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:13.664 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:13.664 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:13.664 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:13.665 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:13.665 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:13.665 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:13.665 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:13.665 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:13.666 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:13.666 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:13.666 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:13.666 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:13.667 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:13.667 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:13.667 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:13.667 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:13.667 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:13.668 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:13.668 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:13.668 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:13.669 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:13.669 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:13.669 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:13.669 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:13.670 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:13.670 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:13.670 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:13.670 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:13.670 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:13.671 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:13.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:13.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:13.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:13.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:13.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:13.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:13.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:13.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:13.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:14.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:14.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:14.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:14.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:14.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:14.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:14.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:14.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:14.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:14.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:14.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:14.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:14.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:15.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12227949 2021-07-26T06:58:15.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12227949 '!=' null 2021-07-26T06:58:15.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:15.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:15.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:15.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:15.110 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:15.111 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:15.111 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:15.111 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:15.111 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:15.111 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:15.112 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:15.112 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:15.112 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:15.112 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:15.112 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:15.113 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:15.113 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:15.113 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:15.113 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:15.113 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:15.114 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:15.114 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:15.114 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:15.114 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:15.115 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:15.115 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:15.115 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:15.115 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:15.115 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:15.116 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:15.116 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:15.116 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:15.116 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:15.116 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:15.117 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:15.117 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:15.117 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:15.117 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:15.117 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:15.118 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:15.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:15.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:15.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:15.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:15.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:15.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:15.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:15.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:15.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:15.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:15.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:15.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:15.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:16.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:16.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:16.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:16.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:16.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:16.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:16.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:16.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:16.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12227955 2021-07-26T06:58:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12227955 '!=' null 2021-07-26T06:58:17.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:17.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:17.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:17.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:17.100 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:17.100 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:17.100 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:17.101 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:17.101 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:17.101 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:17.101 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:17.101 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:17.102 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:17.102 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:17.102 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:17.102 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:17.103 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:17.103 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:17.103 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:17.103 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:17.104 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:17.105 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:17.105 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:17.105 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:17.105 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:17.106 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:17.106 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:17.106 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:17.106 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:17.107 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:17.107 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:17.107 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:17.107 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:17.107 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:17.108 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:17.108 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:17.108 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:17.108 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:17.108 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:17.109 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:17.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:17.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:17.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:17.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:17.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:17.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:17.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:17.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:17.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:17.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:17.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:17.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:18.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:18.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:18.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:18.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:18.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:18.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:18.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:18.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:18.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:18.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18342003 2021-07-26T06:58:18.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18342003 '!=' null 2021-07-26T06:58:18.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:18.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:18.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:18.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:18.638 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:18.638 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:18.638 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:18.639 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:18.639 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:18.639 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:18.639 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:18.640 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:18.640 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:18.640 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:18.640 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:18.641 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:18.641 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:18.641 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:18.641 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:18.641 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:18.642 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:18.642 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:18.642 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:18.642 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:18.643 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:18.643 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:18.643 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:18.643 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:18.644 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:18.644 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:18.644 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:18.645 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:18.645 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:18.645 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:18.645 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:18.646 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:18.646 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:18.646 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:18.646 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:18.646 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:18.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:18.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:18.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:18.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:18.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:18.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:18.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:18.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:18.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:19.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:19.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:19.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:19.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:19.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:19.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:19.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:19.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:19.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:19.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:19.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:19.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:19.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:20.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103307 2021-07-26T06:58:20.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103307 '!=' null 2021-07-26T06:58:20.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:20.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:20.091 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:20.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:20.167 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:20.167 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:20.167 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:20.168 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:20.168 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:20.168 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:20.168 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:20.169 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:20.169 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:20.169 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:20.169 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:20.170 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:20.170 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:20.170 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:20.170 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:20.171 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:20.171 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:20.171 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:20.171 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:20.171 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:20.172 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:20.172 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:20.172 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:20.172 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:20.173 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:20.173 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:20.173 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:20.173 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:20.173 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:20.174 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:20.174 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:20.174 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:20.174 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:20.174 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:20.175 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:20.175 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:20.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:20.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:20.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:20.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:20.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:20.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:20.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:20.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:20.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:20.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:20.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:20.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:20.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:21.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:21.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:21.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:21.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:21.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:21.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:21.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:21.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:21.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:21.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218570 2021-07-26T06:58:21.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218570 '!=' null 2021-07-26T06:58:21.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:21.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:21.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:21.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:22.013 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:22.013 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:22.013 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:22.013 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:22.014 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:22.014 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:22.014 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:22.014 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:22.015 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:22.015 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:22.015 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:22.015 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:22.015 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:22.016 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:22.016 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:22.016 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:22.017 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:22.017 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:22.017 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:22.017 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:22.017 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:22.018 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:22.018 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:22.018 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:22.018 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:22.019 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:22.019 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:22.019 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:22.019 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:22.019 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:22.020 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:22.020 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:22.020 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:22.020 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:22.020 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:22.021 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:22.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:22.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:22.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:22.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:22.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:22.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:22.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:22.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:22.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:22.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:22.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:22.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:22.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:22.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:22.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:22.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:22.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:22.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:22.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:22.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:22.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:22.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:23.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229941 2021-07-26T06:58:23.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229941 '!=' null 2021-07-26T06:58:23.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:23.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:23.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:23.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:23.606 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:23.606 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:23.607 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:23.607 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:23.607 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:23.607 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:23.607 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:23.608 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:23.608 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:23.608 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:23.608 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:23.609 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:23.609 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:23.609 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:23.609 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:23.609 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:23.610 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:23.610 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:23.610 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:23.610 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:23.610 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:23.611 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:23.611 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:23.611 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:23.611 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:23.611 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:23.612 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:23.612 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:23.612 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:23.612 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:23.612 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:23.613 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:23.613 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:23.613 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:23.613 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:23.613 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:23.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:23.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:23.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:23.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:23.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:23.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:23.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:23.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:23.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:24.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:24.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:24.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:24.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:24.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:24.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:24.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:24.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:24.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:24.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:24.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:24.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:24.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:25.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229941 2021-07-26T06:58:25.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229941 '!=' null 2021-07-26T06:58:25.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:25.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:25.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:25.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:25.211 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:25.211 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:25.211 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:25.212 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:25.212 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:25.212 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:25.212 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:25.213 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:25.214 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:25.214 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:25.215 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:25.215 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:25.215 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:25.216 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:25.216 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:25.216 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:25.217 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:25.217 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:25.218 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:25.218 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:25.218 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:25.219 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:25.220 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:25.220 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:25.220 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:25.221 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:25.221 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:25.221 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:25.221 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:25.222 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:25.222 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:25.223 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:25.224 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:25.225 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:25.225 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:25.225 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:25.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:25.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:25.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:25.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:25.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:25.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:25.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:25.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:25.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:26.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:26.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:26.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:26.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:26.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:26.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:26.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:26.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:26.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:26.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:26.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:26.664 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:26.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:27.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229681 2021-07-26T06:58:27.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229681 '!=' null 2021-07-26T06:58:27.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:27.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:27.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:27.199 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:27.280 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:27.281 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:27.281 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:27.281 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:27.281 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:27.282 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:27.282 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:27.282 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:27.282 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:27.288 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:27.289 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:27.289 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:27.289 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:27.289 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:27.290 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:27.290 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:27.290 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:27.290 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:27.290 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:27.291 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:27.291 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:27.291 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:27.291 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:27.292 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:27.292 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:27.292 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:27.292 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:27.292 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:27.293 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:27.293 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:27.293 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:27.293 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:27.293 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:27.294 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:27.294 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:27.294 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:27.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:27.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:27.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:27.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:27.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:27.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:27.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:27.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:27.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:27.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:27.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:27.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:27.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:28.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:28.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:28.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:28.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:28.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:28.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:28.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:28.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:28.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:28.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471027 2021-07-26T06:58:28.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471027 '!=' null 2021-07-26T06:58:28.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:28.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:28.892 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:28.893 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:28.971 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:28.971 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:28.972 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:28.972 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:28.972 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:28.972 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:28.973 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:28.973 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:28.974 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:28.974 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:28.974 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:28.975 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:28.975 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:28.975 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:28.975 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:28.976 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:28.976 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:28.976 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:28.976 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:28.977 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:28.977 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:28.977 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:28.977 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:28.977 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:28.978 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:28.978 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:28.978 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:28.978 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:28.979 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:28.979 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:28.979 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:28.979 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:28.980 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:28.980 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:28.980 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:28.980 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:28.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:29.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:29.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:29.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:29.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:29.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:29.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:29.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:29.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:29.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:29.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:29.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:29.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:29.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:29.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:29.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:29.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:29.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:29.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:29.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:29.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:29.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:30.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356184 2021-07-26T06:58:30.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356184 '!=' null 2021-07-26T06:58:30.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:30.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:30.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:30.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:30.548 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:30.548 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:30.549 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:30.549 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:30.549 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:30.549 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:30.550 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:30.550 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:30.550 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:30.550 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:30.551 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:30.551 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:30.551 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:30.551 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:30.551 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:30.552 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:30.552 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:30.552 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:30.552 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:30.553 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:30.553 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:30.553 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:30.553 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:30.554 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:30.554 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:30.554 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:30.555 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:30.555 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:30.556 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:30.556 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:30.556 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:30.557 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:30.557 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:30.557 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:30.557 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:30.557 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:30.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:30.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:30.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:30.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:30.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:30.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:30.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:30.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:30.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:31.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:31.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:31.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:31.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:31.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:31.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:31.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:31.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:31.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:31.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:31.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:31.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:31.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:32.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471353 2021-07-26T06:58:32.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471353 '!=' null 2021-07-26T06:58:32.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:32.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:32.449 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:32.449 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:32.533 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:32.534 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:32.534 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:32.535 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:32.535 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:32.535 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:32.535 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:32.536 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:32.536 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:32.536 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:32.536 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:32.537 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:32.537 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:32.537 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:32.537 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:32.537 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:32.538 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:32.538 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:32.538 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:32.538 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:32.539 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:32.539 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:32.539 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:32.540 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:32.540 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:32.540 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:32.540 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:32.540 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:32.541 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:32.541 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:32.541 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:32.541 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:32.542 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:32.542 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:32.542 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:32.543 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:32.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:32.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:32.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:32.644 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:32.644 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:32.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:32.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:32.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:32.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:33.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:33.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:33.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:33.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:33.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:33.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:33.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:33.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:33.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:33.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:33.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:33.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:33.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:34.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356364 2021-07-26T06:58:34.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356364 '!=' null 2021-07-26T06:58:34.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:34.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:34.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:34.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:34.144 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:34.144 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:34.144 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:34.145 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:34.145 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:34.145 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:34.145 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:34.146 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:34.146 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:34.147 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:34.147 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:34.148 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:34.148 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:34.148 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:34.148 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:34.149 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:34.149 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:34.149 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:34.149 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:34.150 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:34.150 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:34.150 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:34.150 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:34.151 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:34.151 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:34.151 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:34.151 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:34.151 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:34.152 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:34.152 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:34.152 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:34.152 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:34.153 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:34.153 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:34.154 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:34.154 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:34.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:34.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:34.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:34.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:34.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:34.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:34.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:34.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:34.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:34.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:34.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:34.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:34.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:35.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:35.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:35.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:35.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:35.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:35.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:35.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:35.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:35.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:35.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356414 2021-07-26T06:58:35.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356414 '!=' null 2021-07-26T06:58:35.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:35.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:35.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:35.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:35.620 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:35.621 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:35.621 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:35.621 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:35.622 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:35.622 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:35.622 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:35.622 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:35.622 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:35.623 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:35.623 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:35.623 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:35.623 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:35.623 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:35.624 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:35.624 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:35.624 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:35.624 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:35.624 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:35.625 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:35.625 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:35.625 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:35.625 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:35.626 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:35.626 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:35.626 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:35.626 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:35.627 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:35.627 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:35.627 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:35.627 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:35.627 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:35.628 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:35.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:35.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:35.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:35.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:35.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:35.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:35.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:35.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:35.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:36.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:36.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:36.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:36.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:36.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:36.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:36.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:36.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:36.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:36.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:36.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:36.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:36.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:37.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356414 2021-07-26T06:58:37.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356414 '!=' null 2021-07-26T06:58:37.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:37.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:37.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:37.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:37.262 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:37.262 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:37.262 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:37.263 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:37.263 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:37.263 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:37.263 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:37.264 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:37.264 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:37.264 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:37.264 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:37.264 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:37.265 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:37.265 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:37.265 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:37.265 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:37.266 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:37.266 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:37.266 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:37.266 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:37.266 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:37.267 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:37.267 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:37.267 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:37.267 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:37.267 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:37.268 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:37.268 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:37.268 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:37.268 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:37.268 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:37.269 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:37.269 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:37.269 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:37.270 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:37.270 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:37.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:37.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:37.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:37.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:37.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:37.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:37.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:37.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:37.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:37.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:37.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:37.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:37.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:38.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:38.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:38.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:38.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:38.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:38.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:38.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:38.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:38.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:38.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471765 2021-07-26T06:58:38.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471765 '!=' null 2021-07-26T06:58:38.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:38.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:38.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:38.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:38.797 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:38.797 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:38.797 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:38.798 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:38.798 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:38.798 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:38.798 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:38.799 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:38.799 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:38.799 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:38.799 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:38.799 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:38.800 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:38.800 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:38.800 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:38.800 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:38.801 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:38.801 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:38.801 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:38.801 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:38.801 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:38.802 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:38.802 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:38.802 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:38.802 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:38.802 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:38.803 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:38.803 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:38.803 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:38.803 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:38.804 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:38.804 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:38.804 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:38.804 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:38.804 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:38.804 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:38.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:38.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:38.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:38.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:38.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:38.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:38.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:38.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:38.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:39.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:39.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:39.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:39.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:39.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:39.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:39.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:39.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:39.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:39.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:39.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:39.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:40.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12230100 2021-07-26T06:58:40.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12230100 '!=' null 2021-07-26T06:58:40.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:40.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:40.355 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:40.355 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:40.431 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:40.432 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:40.432 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:40.433 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:40.433 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:40.434 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:40.434 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:40.434 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:40.434 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:40.435 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:40.435 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:40.436 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:40.436 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:40.436 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:40.437 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:40.437 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:40.437 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:40.438 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:40.438 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:40.438 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:40.438 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:40.439 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:40.439 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:40.439 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:40.439 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:40.440 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:40.440 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:40.440 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:40.440 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:40.441 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:40.441 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:40.441 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:40.441 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:40.442 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:40.442 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:40.442 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:40.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:40.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:40.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:40.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:40.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:40.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:40.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:40.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:40.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:40.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:40.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:40.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:40.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:41.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:41.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:41.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:41.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:41.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:41.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:41.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:41.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:41.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:41.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345094 2021-07-26T06:58:41.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345094 '!=' null 2021-07-26T06:58:41.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:41.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:41.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:41.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:42.036 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:42.036 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:42.037 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:42.037 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:42.037 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:42.037 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:42.038 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:42.038 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:42.038 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:42.038 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:42.038 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:42.039 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:42.039 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:42.039 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:42.039 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:42.040 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:42.040 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:42.040 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:42.040 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:42.041 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:42.041 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:42.041 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:42.041 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:42.041 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:42.042 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:42.043 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:42.043 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:42.043 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:42.043 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:42.044 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:42.044 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:42.044 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:42.044 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:42.045 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:42.045 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:42.045 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:42.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:42.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:42.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:42.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:42.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:42.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:42.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:42.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:42.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:42.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:42.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:42.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:42.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:43.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:43.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:43.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:43.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:43.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:43.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:43.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:43.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:43.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:43.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229656 2021-07-26T06:58:43.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229656 '!=' null 2021-07-26T06:58:43.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:43.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:43.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:43.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:43.565 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:43.565 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:43.565 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:43.566 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:43.566 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:43.566 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:43.567 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:43.567 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:43.567 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:43.567 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:43.567 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:43.568 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:43.568 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:43.568 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:43.568 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:43.569 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:43.569 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:43.569 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:43.569 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:43.570 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:43.570 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:43.570 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:43.570 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:43.571 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:43.571 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:43.571 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:43.571 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:43.572 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:43.572 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:43.572 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:43.573 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:43.573 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:43.573 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:43.573 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:43.574 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:43.574 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:43.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:43.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:43.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:43.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:43.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:43.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:43.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:43.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:43.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:44.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:44.218 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:44.218 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:44.218 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:44.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:44.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:44.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:44.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:44.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:44.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:44.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:44.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:44.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:45.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229656 2021-07-26T06:58:45.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229656 '!=' null 2021-07-26T06:58:45.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:45.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:45.217 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:45.217 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:45.287 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:45.287 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:45.288 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:45.288 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:45.288 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:45.288 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:45.289 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:45.289 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:45.289 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:45.289 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:45.290 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:45.290 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:45.290 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:45.290 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:45.291 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:45.291 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:45.291 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:45.291 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:45.291 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:45.292 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:45.292 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:45.292 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:45.292 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:45.292 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:45.293 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:45.293 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:45.293 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:45.293 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:45.293 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:45.294 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:45.294 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:45.294 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:45.294 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:45.295 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:45.295 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:45.295 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:45.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:45.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:45.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:45.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:45.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:45.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:45.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:45.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:45.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:45.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:45.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:45.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:45.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:46.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:46.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:46.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:46.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:46.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:46.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:46.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:46.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:46.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:46.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229452 2021-07-26T06:58:46.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229452 '!=' null 2021-07-26T06:58:46.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:46.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:46.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:46.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:47.083 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:47.083 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:47.083 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:47.083 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:47.084 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:47.084 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:47.084 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:47.084 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:47.084 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:47.085 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:47.085 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:47.086 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:47.086 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:47.086 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:47.086 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:47.087 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:47.088 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:47.088 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:47.089 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:47.089 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:47.090 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:47.090 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:47.091 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:47.091 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:47.091 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:47.092 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:47.092 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:47.093 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:47.093 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:47.093 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:47.094 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:47.094 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:47.095 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:47.096 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:47.096 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:47.096 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:47.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:47.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:47.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:47.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:47.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:47.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:47.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:47.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:47.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:47.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:47.688 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:47.689 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:47.689 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:48.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:48.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:48.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:48.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:48.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:48.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:48.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:48.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:48.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:48.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344253 2021-07-26T06:58:48.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344253 '!=' null 2021-07-26T06:58:48.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:48.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:48.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:48.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:48.637 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:48.637 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:48.637 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:48.637 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:48.638 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:48.638 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:48.638 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:48.638 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:48.639 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:48.639 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:48.639 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:48.639 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:48.639 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:48.640 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:48.640 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:48.640 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:48.640 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:48.640 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:48.641 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:48.641 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:48.641 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:48.641 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:48.641 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:48.642 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:48.642 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:48.642 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:48.642 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:48.643 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:48.643 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:48.643 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:48.643 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:48.643 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:48.644 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:48.644 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:48.644 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:48.644 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:48.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:48.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:48.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:48.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:48.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:48.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:48.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:48.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:48.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:49.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:49.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:49.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:49.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:49.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:49.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:49.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:49.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:49.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:49.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:49.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:49.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:49.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:50.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229271 2021-07-26T06:58:50.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229271 '!=' null 2021-07-26T06:58:50.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:50.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:50.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:50.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:50.607 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:50.608 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:50.608 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:50.608 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:50.609 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:50.609 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:50.609 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:50.609 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:50.610 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:50.610 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:50.610 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:50.610 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:50.611 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:50.611 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:50.611 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:50.611 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:50.611 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:50.612 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:50.612 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:50.612 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:50.612 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:50.613 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:50.613 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:50.613 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:50.613 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:50.613 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:50.614 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:50.614 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:50.614 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:50.614 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:50.614 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:50.615 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:50.615 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:50.615 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:50.615 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:50.616 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:50.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:50.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:50.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:50.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:50.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:50.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:50.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:50.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:50.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:51.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:51.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:51.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:51.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:51.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:51.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:51.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:51.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:51.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:51.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:51.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:51.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:51.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:52.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091396 2021-07-26T06:58:52.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091396 '!=' null 2021-07-26T06:58:52.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:52.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:52.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:52.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:52.334 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:52.335 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:52.335 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:52.335 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:52.336 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:52.336 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:52.336 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:52.337 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:52.337 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:52.338 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:52.338 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:52.339 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:52.339 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:52.339 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:52.340 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:52.340 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:52.341 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:52.341 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:52.341 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:52.342 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:52.342 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:52.343 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:52.343 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:52.343 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:52.344 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:52.344 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:52.344 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:52.345 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:52.345 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:52.345 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:52.345 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:52.345 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:52.346 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:52.346 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:52.346 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:52.346 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:52.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:52.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:52.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:52.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:52.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:52.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:52.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:52.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:52.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:52.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:52.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:52.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:52.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:53.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:53.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:53.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:53.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:53.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:53.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:53.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:53.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:53.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:53.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976618 2021-07-26T06:58:53.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976618 '!=' null 2021-07-26T06:58:53.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:53.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:53.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:53.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:53.978 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:53.979 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:53.979 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:53.979 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:53.979 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:53.979 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:53.980 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:53.980 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:53.980 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:53.981 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:53.981 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:53.981 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:53.981 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:53.981 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:53.982 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:53.982 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:53.982 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:53.982 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:53.983 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:53.983 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:53.983 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:53.983 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:53.984 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:53.984 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:53.984 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:53.984 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:53.984 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:53.985 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:53.985 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:53.985 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:53.985 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:53.985 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:53.986 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:53.986 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:53.986 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:53.986 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:53.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:54.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:54.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:54.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:54.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:54.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:54.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:54.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:54.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:54.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:54.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:54.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:54.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:54.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:54.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:54.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:54.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:54.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:54.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:54.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:54.890 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:54.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:55.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976717 2021-07-26T06:58:55.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976717 '!=' null 2021-07-26T06:58:55.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:55.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:55.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:55.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:55.578 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:55.578 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:55.579 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:55.579 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:55.579 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:55.579 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:55.580 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:55.580 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:55.580 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:55.580 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:55.581 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:55.581 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:55.581 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:55.582 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:55.582 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:55.583 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:55.583 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:55.583 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:55.583 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:55.587 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:55.587 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:55.587 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:55.587 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:55.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:55.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:55.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:55.690 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:55.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:55.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:55.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:55.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:55.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:56.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:56.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:56.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:56.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:56.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:56.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:56.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:56.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:56.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:56.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:56.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:56.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:56.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:57.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976717 2021-07-26T06:58:57.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976717 '!=' null 2021-07-26T06:58:57.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:57.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:57.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:57.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:57.295 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:57.295 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:57.296 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:57.296 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:57.297 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:57.297 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:57.298 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:57.298 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:57.298 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:57.299 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:57.299 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:57.299 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:57.300 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:57.300 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:57.301 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:57.301 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:57.301 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:57.302 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:57.302 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:57.302 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:57.303 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:57.303 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:57.304 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:57.304 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:57.304 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:57.305 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:57.305 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:57.305 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:57.305 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:57.305 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:57.306 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:57.306 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:57.306 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:57.306 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:57.306 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:57.307 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:57.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:57.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:57.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:57.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:57.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:57.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:57.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:57.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:57.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:57.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:57.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:57.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:57.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:58:58.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:58:58.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:58:58.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:58:58.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:58:58.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:58:58.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:58:58.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:58:58.311 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:58:58.311 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:58:58.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091666 2021-07-26T06:58:58.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091666 '!=' null 2021-07-26T06:58:58.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:58:58.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:58.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:58:58.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:58:58.900 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:58:58.900 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:58:58.900 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:58.901 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:58.901 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:58:58.901 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:58.901 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:58:58.902 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:58:58.902 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:58.902 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:58:58.902 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:58.902 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:58.903 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:58.903 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:58.903 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:58:58.903 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:58:58.903 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:58.904 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:58.904 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:58.904 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:58.904 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:58:58.905 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:58:58.905 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:58:58.905 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:58:58.905 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:58:58.905 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:58:58.906 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:58.906 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:58:58.906 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:58:58.906 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:58:58.906 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:58:58.907 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:58:58.907 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:58:58.907 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:58:58.907 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:58:58.907 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:58:58.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:58:59.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:58:59.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:58:59.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:58:59.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:58:59.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:58:59.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:58:59.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:58:59.013 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:58:59.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:58:59.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:58:59.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:58:59.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:00.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:00.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:00.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:00.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:00.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:00.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:00.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:00.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:00.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:00.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976558 2021-07-26T06:59:00.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976558 '!=' null 2021-07-26T06:59:00.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:00.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:00.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:00.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:00.723 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:00.723 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:00.723 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:00.724 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:00.725 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:00.725 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:00.725 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:00.725 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:00.725 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:00.726 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:00.726 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:00.726 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:00.726 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:00.726 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:00.727 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:00.727 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:00.727 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:00.727 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:00.728 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:00.728 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:00.728 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:00.728 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:00.728 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:00.729 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:00.729 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:00.729 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:00.729 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:00.729 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:00.730 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:00.730 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:00.730 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:00.731 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:00.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:00.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:00.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:00.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:00.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:00.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:00.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:00.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:00.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:01.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:01.225 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:01.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:01.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:01.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:01.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:01.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:01.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:01.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:01.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:01.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:01.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:01.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:02.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965071 2021-07-26T06:59:02.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965071 '!=' null 2021-07-26T06:59:02.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:02.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:02.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:02.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:02.487 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:02.487 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:02.487 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:02.488 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:02.488 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:02.488 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:02.488 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:02.489 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:02.489 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:02.489 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:02.489 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:02.489 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:02.490 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:02.490 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:02.490 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:02.490 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:02.491 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:02.491 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:02.491 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:02.491 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:02.492 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:02.492 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:02.492 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:02.492 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:02.492 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:02.493 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:02.493 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:02.493 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:02.493 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:02.494 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:02.494 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:02.494 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:02.494 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:02.494 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:02.495 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:02.495 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:02.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:02.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:02.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:02.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:02.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:02.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:02.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:02.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:02.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:03.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:03.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:03.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:03.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:03.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:03.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:03.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:03.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:03.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:03.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:03.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:03.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:03.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:03.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102987 2021-07-26T06:59:03.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102987 '!=' null 2021-07-26T06:59:03.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:03.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:03.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:03.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:04.002 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:04.003 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:04.003 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:04.003 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:04.003 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:04.004 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:04.004 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:04.004 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:04.004 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:04.005 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:04.005 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:04.005 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:04.005 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:04.005 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:04.006 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:04.006 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:04.007 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:04.007 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:04.007 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:04.008 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:04.008 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:04.008 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:04.008 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:04.008 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:04.009 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:04.009 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:04.009 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:04.009 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:04.010 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:04.010 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:04.010 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:04.010 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:04.011 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:04.011 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:04.011 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:04.012 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:04.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:04.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:04.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:04.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:04.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:04.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:04.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:04.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:04.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:04.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:04.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:04.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:04.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:04.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:04.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:04.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:04.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:04.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:04.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:04.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:04.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:04.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:05.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102987 2021-07-26T06:59:05.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102987 '!=' null 2021-07-26T06:59:05.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:05.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:05.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:05.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:05.790 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:05.790 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:05.791 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:05.791 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:05.791 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:05.791 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:05.792 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:05.792 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:05.792 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:05.792 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:05.793 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:05.793 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:05.793 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:05.793 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:05.794 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:05.794 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:05.794 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:05.794 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:05.795 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:05.795 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:05.795 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:05.795 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:05.795 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:05.796 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:05.796 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:05.796 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:05.796 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:05.796 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:05.797 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:05.797 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:05.797 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:05.797 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:05.798 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:05.798 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:05.798 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:05.798 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:05.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:05.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:05.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:05.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:05.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:05.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:05.902 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:05.902 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:05.902 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:06.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:06.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:06.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:06.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:06.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:06.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:06.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:06.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:06.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:06.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:06.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:06.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:06.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:07.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217734 2021-07-26T06:59:07.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217734 '!=' null 2021-07-26T06:59:07.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:07.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:07.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:07.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:07.561 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:07.561 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:07.562 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:07.562 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:07.562 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:07.562 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:07.563 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:07.563 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:07.563 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:07.563 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:07.563 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:07.564 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:07.564 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:07.564 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:07.564 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:07.565 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:07.565 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:07.565 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:07.565 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:07.565 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:07.569 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:07.569 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:07.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:07.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:07.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:07.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:07.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:07.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:07.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:07.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:07.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:08.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:08.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:08.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:08.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:08.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:08.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:08.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:08.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:08.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:08.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:08.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:08.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:08.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:09.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217734 2021-07-26T06:59:09.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217734 '!=' null 2021-07-26T06:59:09.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:09.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:09.108 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:09.108 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:09.193 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:09.193 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:09.194 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:09.194 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:09.194 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:09.194 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:09.195 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:09.195 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:09.195 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:09.196 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:09.196 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:09.196 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:09.196 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:09.197 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:09.197 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:09.197 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:09.197 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:09.197 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:09.198 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:09.198 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:09.198 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:09.198 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:09.199 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:09.199 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:09.199 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:09.199 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:09.199 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:09.200 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:09.200 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:09.200 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:09.200 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:09.201 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:09.201 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:09.201 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:09.202 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:09.202 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:09.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:09.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:09.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:09.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:09.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:09.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:09.305 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:09.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:09.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:09.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:09.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:09.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:09.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:10.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:10.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:10.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:10.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:10.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:10.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:10.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:10.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:10.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:10.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103054 2021-07-26T06:59:10.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103054 '!=' null 2021-07-26T06:59:10.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:10.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:10.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:10.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:10.697 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:10.698 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:10.698 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:10.699 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:10.699 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:10.699 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:10.699 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:10.699 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:10.700 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:10.700 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:10.700 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:10.700 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:10.701 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:10.701 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:10.701 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:10.701 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:10.701 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:10.702 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:10.702 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:10.702 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:10.702 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:10.702 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:10.703 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:10.703 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:10.703 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:10.703 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:10.704 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:10.704 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:10.704 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:10.704 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:10.704 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:10.705 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:10.705 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:10.705 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:10.705 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:10.705 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:10.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:10.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:10.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:10.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:10.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:10.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:10.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:10.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:10.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:11.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:11.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:11.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:11.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:12.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:12.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:12.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:12.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:12.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:12.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:12.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:12.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:12.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:12.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091543 2021-07-26T06:59:12.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091543 '!=' null 2021-07-26T06:59:12.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:12.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:12.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:12.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:12.692 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:12.693 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:12.693 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:12.693 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:12.694 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:12.694 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:12.694 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:12.694 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:12.694 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:12.695 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:12.695 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:12.695 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:12.696 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:12.696 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:12.696 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:12.696 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:12.696 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:12.697 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:12.697 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:12.697 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:12.697 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:12.697 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:12.698 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:12.698 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:12.698 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:12.698 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:12.698 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:12.699 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:12.699 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:12.699 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:12.699 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:12.700 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:12.700 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:12.701 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:12.701 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:12.701 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:12.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:12.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:12.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:12.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:12.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:12.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:12.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:12.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:12.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:13.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:13.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:13.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:13.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:13.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:13.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:13.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:13.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:13.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:13.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:13.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:13.745 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:13.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:14.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103003 2021-07-26T06:59:14.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103003 '!=' null 2021-07-26T06:59:14.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:14.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:14.196 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:14.196 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:14.271 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:14.271 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:14.271 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:14.272 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:14.273 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:14.274 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:14.275 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:14.276 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:14.277 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:14.278 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:14.278 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:14.278 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:14.278 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:14.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:14.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:14.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:14.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:14.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:14.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:14.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:14.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:14.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:14.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:14.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:14.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:14.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:15.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:15.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:15.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:15.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:15.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:15.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:15.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:15.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:15.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:15.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102918 2021-07-26T06:59:15.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102918 '!=' null 2021-07-26T06:59:15.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:15.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:15.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:15.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:15.892 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:15.893 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:15.893 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:15.894 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:15.894 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:15.894 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:15.895 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:15.895 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:15.896 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:15.896 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:15.896 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:15.897 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:15.897 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:15.897 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:15.898 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:15.898 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:15.898 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:15.899 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:15.899 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:15.900 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:15.900 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:15.901 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:15.901 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:15.902 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:15.902 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:15.902 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:15.903 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:15.903 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:15.903 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:15.904 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:15.904 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:15.904 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:15.905 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:15.906 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:15.906 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:15.906 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:15.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:16.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:16.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:16.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:16.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:16.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:16.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:16.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:16.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:16.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:16.336 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:16.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:16.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:16.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:16.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:16.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:16.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:16.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:17.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102918 2021-07-26T06:59:17.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102918 '!=' null 2021-07-26T06:59:17.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:17.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:17.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:17.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:17.388 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:17.389 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:17.389 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:17.389 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:17.389 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:17.389 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:17.390 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:17.390 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:17.390 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:17.390 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:17.391 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:17.391 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:17.391 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:17.391 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:17.391 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:17.392 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:17.392 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:17.392 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:17.392 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:17.392 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:17.393 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:17.393 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:17.393 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:17.393 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:17.394 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:17.394 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:17.394 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:17.394 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:17.394 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:17.395 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:17.395 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:17.395 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:17.395 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:17.395 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:17.396 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:17.396 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:17.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:17.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:17.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:17.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:17.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:17.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:17.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:17.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:17.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:17.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:17.861 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:17.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:17.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:18.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:18.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:18.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:18.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:18.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:18.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:18.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:18.431 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:18.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:18.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344562 2021-07-26T06:59:18.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344562 '!=' null 2021-07-26T06:59:18.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:18.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:18.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:18.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:19.017 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:19.017 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:19.017 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:19.017 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:19.018 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:19.018 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:19.018 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:19.019 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:19.019 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:19.019 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:19.019 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:19.020 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:19.020 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:19.020 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:19.021 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:19.021 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:19.021 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:19.021 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:19.022 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:19.022 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:19.022 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:19.022 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:19.022 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:19.023 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:19.023 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:19.023 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:19.023 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:19.024 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:19.024 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:19.024 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:19.024 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:19.024 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:19.025 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:19.025 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:19.025 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:19.025 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:19.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:19.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:19.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:19.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:19.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:19.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:19.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:19.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:19.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:19.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:19.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:19.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:19.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:20.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:20.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:20.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:20.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:20.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:20.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:20.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:20.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:20.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:20.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229514 2021-07-26T06:59:20.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229514 '!=' null 2021-07-26T06:59:20.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:20.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:20.755 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:20.755 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:20.829 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:20.829 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:20.829 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:20.830 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:20.830 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:20.830 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:20.830 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:20.830 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:20.831 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:20.831 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:20.832 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:20.832 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:20.832 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:20.833 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:20.833 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:20.834 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:20.834 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:20.834 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:20.835 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:20.835 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:20.836 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:20.836 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:20.836 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:20.837 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:20.837 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:20.838 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:20.839 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:20.839 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:20.839 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:20.840 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:20.840 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:20.840 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:20.841 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:20.841 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:20.842 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:20.842 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:20.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:20.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:20.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:20.939 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:20.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:20.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:20.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:20.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:20.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:21.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:21.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:21.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:21.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:21.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:21.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:21.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:21.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:21.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:21.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:21.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:21.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:21.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:22.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217769 2021-07-26T06:59:22.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217769 '!=' null 2021-07-26T06:59:22.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:22.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:22.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:22.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:22.401 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:22.402 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:22.402 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:22.402 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:22.403 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:22.403 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:22.403 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:22.403 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:22.404 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:22.404 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:22.404 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:22.404 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:22.404 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:22.405 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:22.405 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:22.405 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:22.405 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:22.406 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:22.406 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:22.406 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:22.406 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:22.406 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:22.407 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:22.407 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:22.407 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:22.407 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:22.408 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:22.408 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:22.408 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:22.408 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:22.408 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:22.409 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:22.409 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:22.409 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:22.409 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:22.410 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:22.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:22.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:22.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:22.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:22.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:22.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:22.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:22.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:22.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:22.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:22.886 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:22.886 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:22.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:23.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:23.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:23.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:23.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:23.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:23.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:23.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:23.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:23.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:23.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229590 2021-07-26T06:59:23.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229590 '!=' null 2021-07-26T06:59:23.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:23.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:23.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:23.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:23.942 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:23.942 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:23.942 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:23.943 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:23.943 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:23.943 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:23.943 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:23.943 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:23.944 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:23.944 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:23.944 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:23.944 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:23.945 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:23.945 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:23.945 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:23.945 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:23.945 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:23.946 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:23.946 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:23.946 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:23.946 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:23.947 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:23.947 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:23.947 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:23.947 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:23.947 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:23.948 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:23.948 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:23.948 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:23.948 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:23.948 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:23.949 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:23.949 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:23.949 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:23.949 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:23.950 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:23.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:24.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:24.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:24.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:24.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:24.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:24.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:24.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:24.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:24.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:24.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:24.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:24.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:24.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:24.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:24.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:24.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:24.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:24.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:24.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:24.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:24.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:25.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229487 2021-07-26T06:59:25.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229487 '!=' null 2021-07-26T06:59:25.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:25.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:25.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:25.429 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:25.513 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:25.513 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:25.513 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:25.514 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:25.514 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:25.514 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:25.514 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:25.515 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:25.515 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:25.515 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:25.515 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:25.515 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:25.516 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:25.516 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:25.516 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:25.516 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:25.517 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:25.517 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:25.517 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:25.518 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:25.518 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:25.518 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:25.518 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:25.519 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:25.519 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:25.519 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:25.519 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:25.519 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:25.520 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:25.520 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:25.520 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:25.520 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:25.520 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:25.521 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:25.521 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:25.521 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:25.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:25.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:25.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:25.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:25.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:25.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:25.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:25.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:25.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:26.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:26.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:26.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:26.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:26.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:26.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:26.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:26.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:26.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:26.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:26.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:26.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:26.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:27.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344421 2021-07-26T06:59:27.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344421 '!=' null 2021-07-26T06:59:27.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:27.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:27.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:27.540 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:27.622 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:27.622 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:27.623 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:27.623 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:27.623 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:27.623 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:27.623 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:27.624 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:27.624 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:27.624 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:27.624 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:27.625 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:27.625 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:27.625 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:27.625 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:27.625 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:27.626 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:27.626 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:27.626 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:27.626 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:27.626 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:27.627 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:27.627 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:27.627 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:27.627 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:27.628 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:27.628 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:27.628 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:27.628 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:27.628 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:27.629 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:27.629 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:27.629 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:27.629 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:27.629 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:27.630 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:27.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:27.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:27.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:27.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:27.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:27.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:27.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:27.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:27.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:28.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:28.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:28.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:28.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:28.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:28.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:28.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:28.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:28.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:28.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:28.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:28.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:28.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:29.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344421 2021-07-26T06:59:29.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344421 '!=' null 2021-07-26T06:59:29.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:29.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:29.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:29.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:29.200 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:29.201 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:29.201 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:29.201 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:29.201 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:29.201 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:29.202 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:29.202 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:29.202 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:29.202 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:29.203 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:29.203 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:29.203 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:29.203 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:29.203 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:29.204 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:29.204 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:29.204 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:29.204 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:29.205 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:29.205 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:29.205 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:29.205 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:29.205 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:29.206 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:29.206 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:29.206 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:29.206 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:29.207 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:29.207 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:29.207 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:29.207 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:29.207 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:29.208 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:29.208 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:29.208 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:29.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:29.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:29.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:29.311 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:29.311 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:29.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:29.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:29.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:29.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:29.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:29.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:29.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:29.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:30.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:30.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:30.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:30.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:30.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:30.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:30.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:30.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:30.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:31.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103086 2021-07-26T06:59:31.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103086 '!=' null 2021-07-26T06:59:31.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:31.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:31.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:31.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:31.237 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:31.238 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:31.238 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:31.238 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:31.238 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:31.239 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:31.239 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:31.239 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:31.239 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:31.239 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:31.240 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:31.240 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:31.240 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:31.240 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:31.241 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:31.241 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:31.241 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:31.242 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:31.242 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:31.242 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:31.242 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:31.242 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:31.243 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:31.243 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:31.243 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:31.243 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:31.243 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:31.244 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:31.244 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:31.244 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:31.244 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:31.244 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:31.245 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:31.245 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:31.245 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:31.245 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:31.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:31.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:31.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:31.347 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:31.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:31.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:31.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:31.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:31.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:31.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:31.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:31.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:31.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:32.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:32.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:32.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:32.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:32.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:32.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:32.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:32.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:32.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:32.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17471311 2021-07-26T06:59:32.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17471311 '!=' null 2021-07-26T06:59:32.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:32.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:32.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:32.803 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:32.873 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:32.874 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:32.874 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:32.874 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:32.874 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:32.875 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:32.875 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:32.875 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:32.875 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:32.875 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:32.876 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:32.876 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:32.876 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:32.876 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:32.876 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:32.877 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:32.877 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:32.877 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:32.877 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:32.878 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:32.878 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:32.878 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:32.878 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:32.879 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:32.879 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:32.879 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:32.879 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:32.879 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:32.880 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:32.880 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:32.880 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:32.880 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:32.880 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:32.881 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:32.881 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:32.881 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:32.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:32.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:32.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:32.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:32.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:32.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:32.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:32.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:32.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:33.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:33.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:33.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:33.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:33.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:33.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:33.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:33.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:33.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:33.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:33.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:33.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:33.902 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:34.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482534 2021-07-26T06:59:34.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482534 '!=' null 2021-07-26T06:59:34.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:34.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:34.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:34.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:34.559 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:34.559 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:34.559 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:34.560 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:34.560 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:34.560 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:34.561 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:34.561 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:34.561 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:34.561 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:34.561 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:34.562 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:34.562 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:34.562 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:34.562 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:34.563 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:34.563 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:34.563 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:34.563 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:34.563 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:34.564 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:34.564 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:34.564 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:34.564 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "7.0", 2021-07-26T06:59:34.564 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 255, 2021-07-26T06:59:34.565 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:34.565 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:34.565 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:34.565 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:34.565 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:34.566 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:34.566 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:34.566 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:34.566 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:34.567 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:34.567 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:34.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:34.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:34.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:34.669 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:34.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:34.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:34.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:34.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:34.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:35.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:35.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:35.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:35.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:35.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:35.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:35.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:35.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:35.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:35.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:35.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:35.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:35.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:36.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12579098 2021-07-26T06:59:36.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12579098 '!=' null 2021-07-26T06:59:36.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:36.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:36.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:36.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:36.698 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:36.698 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:36.699 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:36.699 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:36.699 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:36.699 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:36.700 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:36.700 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:36.700 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:36.700 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:36.700 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:36.701 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:36.701 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:36.701 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:36.701 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:36.702 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:36.702 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:36.702 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:36.702 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:36.702 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:36.703 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:36.703 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:36.703 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:36.703 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:36.704 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:36.704 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:36.704 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:36.704 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:36.704 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:36.705 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:36.705 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:36.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:36.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:36.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:36.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:36.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:36.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:36.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:36.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:36.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:37.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:37.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:37.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:37.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:37.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:37.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:37.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:37.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:37.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:37.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:37.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:37.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:37.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:38.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14413340 2021-07-26T06:59:38.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14413340 '!=' null 2021-07-26T06:59:38.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:38.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:38.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:38.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:38.498 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:38.499 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:38.499 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:38.499 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:38.499 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:38.500 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:38.500 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:38.501 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:38.501 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:38.501 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:38.502 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:38.503 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:38.504 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:38.504 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:38.504 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:38.504 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:38.505 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:38.505 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:38.505 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:38.505 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:38.505 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:38.506 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:38.506 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:38.506 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:38.507 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:38.507 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:38.507 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:38.507 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:38.507 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:38.508 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:38.508 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:38.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:38.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:38.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:38.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:38.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:38.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:38.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:38.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:38.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:38.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T06:59:38.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:38.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:38.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:39.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 3 2021-07-26T06:59:39.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T06:59:39.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:39.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:39.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:39.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:39.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:39.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:39.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:40.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7861774 2021-07-26T06:59:40.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7861774 '!=' null 2021-07-26T06:59:40.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:40.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:40.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:40.641 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:40.721 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:40.721 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:40.722 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:40.722 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:40.722 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:40.722 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:40.723 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:40.723 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:40.723 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:40.723 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:40.724 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:40.724 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:40.724 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:40.724 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:40.724 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:40.725 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:40.725 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:40.726 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:40.727 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:40.727 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:40.727 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:40.727 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:40.728 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:40.728 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:40.728 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:40.728 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:40.728 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:40.729 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:40.729 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:40.729 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:40.729 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:40.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:40.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:40.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:40.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:40.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:40.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:40.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:40.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:40.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:41.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:41.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:41.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:41.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:41.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:41.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 0 2021-07-26T06:59:41.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T06:59:41.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=1 2021-07-26T06:59:41.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:41.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:41.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:41.870 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:41.870 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:41.870 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:41.870 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:41.871 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:41.871 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:41.871 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:41.871 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:41.872 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:41.872 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:41.872 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:41.872 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:41.872 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:41.873 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:41.873 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:41.874 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:41.874 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:41.874 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:41.874 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:41.874 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:41.875 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:41.875 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:41.875 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:41.875 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:41.876 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:41.876 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:41.876 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:41.876 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:41.876 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:41.877 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:41.877 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:41.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:41.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:41.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:41.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:41.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:41.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:41.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:41.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:41.983 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:42.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:42.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:42.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:42.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:42.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:42.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:42.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:42.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:42.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:42.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:42.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:42.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:42.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:43.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16292545 2021-07-26T06:59:43.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16292545 '!=' null 2021-07-26T06:59:43.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:43.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:43.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:43.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:43.622 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:43.623 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:43.623 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:43.623 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:43.624 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:43.624 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:43.624 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:43.624 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:43.625 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:43.625 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:43.625 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:43.625 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:43.625 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:43.626 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:43.626 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:43.627 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:43.627 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:43.627 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:43.628 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:43.628 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:43.628 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:43.628 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:43.628 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:43.629 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:43.629 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:43.629 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:43.629 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:43.630 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:43.630 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:43.630 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:43.630 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:43.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:43.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:43.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:43.733 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:43.733 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:43.733 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:43.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:43.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:43.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:44.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:44.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:44.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:44.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:44.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:44.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:44.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:44.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:44.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:44.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:44.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:44.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:44.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:45.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16292545 2021-07-26T06:59:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16292545 '!=' null 2021-07-26T06:59:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:45.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:45.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:45.333 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:45.333 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:45.334 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:45.334 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:45.334 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:45.335 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:45.335 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:45.335 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:45.335 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:45.335 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:45.336 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:45.336 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:45.337 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:45.337 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:45.337 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:45.337 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:45.338 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:45.338 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:45.338 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:45.338 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:45.339 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:45.339 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:45.339 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:45.339 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:45.339 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:45.340 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:45.340 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:45.340 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:45.340 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:45.341 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:45.341 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:45.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:45.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:45.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:45.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:45.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:45.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:45.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:45.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:45.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:45.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:45.854 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:45.855 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:45.855 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:46.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:46.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:46.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:46.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:46.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:46.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:46.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:46.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:46.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:46.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13627737 2021-07-26T06:59:46.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13627737 '!=' null 2021-07-26T06:59:46.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:46.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:46.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:46.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:47.046 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:47.046 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:47.046 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:47.046 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:47.047 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:47.047 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:47.047 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:47.047 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:47.047 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:47.048 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:47.048 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:47.048 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:47.049 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:47.049 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:47.049 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:47.049 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:47.050 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:47.050 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:47.050 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:47.050 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:47.050 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:47.051 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:47.051 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:47.051 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:47.051 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:47.051 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:47.052 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:47.052 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:47.052 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:47.052 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:47.053 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:47.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:47.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:47.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:47.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:47.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:47.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:47.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:47.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:47.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:47.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:47.642 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:47.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:47.643 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:48.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:48.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:48.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:48.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:48.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:48.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:48.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:48.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:48.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:48.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20705771 2021-07-26T06:59:48.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20705771 '!=' null 2021-07-26T06:59:48.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:48.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:48.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:48.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:48.701 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:48.701 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:48.702 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:48.702 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:48.703 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:48.703 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:48.703 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:48.704 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:48.704 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:48.705 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:48.705 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:48.705 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:48.706 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:48.706 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:48.710 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:48.710 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:48.710 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:48.710 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:48.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:48.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:48.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:48.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:48.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:48.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:48.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:48.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:48.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:49.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:49.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:49.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:49.286 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:49.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:49.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:49.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:49.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:49.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:49.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:49.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:49.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:49.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:50.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14850535 2021-07-26T06:59:50.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14850535 '!=' null 2021-07-26T06:59:50.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:50.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:50.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:50.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:50.721 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:50.721 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:50.721 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:50.721 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:50.722 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:50.722 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:50.722 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:50.722 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:50.723 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:50.723 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:50.724 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:50.724 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:50.724 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:50.724 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:50.725 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:50.725 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:50.725 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:50.726 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:50.726 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:50.726 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:50.727 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:50.727 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:50.727 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:50.727 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:50.727 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:50.728 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:50.728 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:50.728 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:50.728 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:50.729 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:50.729 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:50.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:50.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:50.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:50.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:50.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:50.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:50.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:50.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:50.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:51.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:51.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:51.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:51.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:52.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:52.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:52.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:52.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:52.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:52.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:52.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:52.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:52.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:52.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965196 2021-07-26T06:59:52.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965196 '!=' null 2021-07-26T06:59:52.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:52.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:52.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:52.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:52.811 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:52.812 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:52.812 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:52.812 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:52.813 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:52.813 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:52.813 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:52.813 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:52.813 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:52.814 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:52.814 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:52.814 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:52.815 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:52.815 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:52.815 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:52.815 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:52.815 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:52.816 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:52.816 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:52.816 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:52.816 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:52.816 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:52.817 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:52.817 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:52.817 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:52.818 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:52.818 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:52.818 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:52.818 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:52.819 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:52.819 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:52.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:52.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:52.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:52.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:52.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:52.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:52.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:52.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:52.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:53.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:53.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:53.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:53.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:53.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:53.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:53.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:53.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:53.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:53.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:53.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:53.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:53.747 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:54.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14850392 2021-07-26T06:59:54.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14850392 '!=' null 2021-07-26T06:59:54.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:54.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:54.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:54.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:54.363 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:54.364 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:54.364 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:54.364 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:54.364 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:54.364 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:54.365 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:54.365 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:54.365 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:54.365 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:54.366 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:54.366 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:54.366 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:54.366 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:54.366 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:54.367 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:54.367 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:54.367 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:54.367 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:54.367 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:54.368 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:54.368 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:54.368 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:54.368 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:54.368 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:54.369 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:54.369 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:54.369 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:54.369 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:54.370 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:54.370 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:54.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:54.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:54.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:54.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:54.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:54.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:54.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:54.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:54.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:54.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:54.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:54.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:54.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:55.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:55.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:55.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:55.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:55.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:55.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:55.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:55.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:55.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:55.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14850185 2021-07-26T06:59:55.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14850185 '!=' null 2021-07-26T06:59:55.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:55.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:55.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:55.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:55.883 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:55.884 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:55.884 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:55.884 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:55.885 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:55.885 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:55.885 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:55.886 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:55.886 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:55.886 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:55.886 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:55.887 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:55.887 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:55.887 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:55.887 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:55.887 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:55.888 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:55.888 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:55.888 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:55.888 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:55.888 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:55.889 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:55.889 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:55.889 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:55.889 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:55.889 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:55.890 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:55.890 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:55.891 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:55.891 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:55.891 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:55.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:55.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:55.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:55.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:55.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:55.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:55.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:55.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:55.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:56.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:56.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:56.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:56.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:56.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:56.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:56.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:56.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:56.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:56.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:56.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:56.803 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:56.803 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:57.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14850185 2021-07-26T06:59:57.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14850185 '!=' null 2021-07-26T06:59:57.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:57.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:57.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:57.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:57.461 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:57.461 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:57.461 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:57.461 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:57.462 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:57.462 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:57.462 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:57.462 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:57.462 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:57.463 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:57.463 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:57.463 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:57.463 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:57.464 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:57.464 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:57.464 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:57.464 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:57.464 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:57.465 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:57.465 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:57.465 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:57.465 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:57.465 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:57.466 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:57.466 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:57.466 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:57.466 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:57.467 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:57.467 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:57.467 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:57.467 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:57.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:57.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:57.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:57.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:57.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:57.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:57.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:57.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:57.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:58.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:58.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:58.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:58.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T06:59:58.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T06:59:58.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T06:59:58.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T06:59:58.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T06:59:58.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T06:59:58.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T06:59:58.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T06:59:58.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T06:59:58.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T06:59:59.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=22711550 2021-07-26T06:59:59.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 22711550 '!=' null 2021-07-26T06:59:59.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T06:59:59.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:59.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T06:59:59.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T06:59:59.104 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T06:59:59.104 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T06:59:59.105 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:59.105 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:59.105 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T06:59:59.105 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:59.106 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T06:59:59.106 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T06:59:59.106 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:59.106 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T06:59:59.107 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:59.107 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:59.107 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:59.107 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:59.107 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:59.108 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T06:59:59.108 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T06:59:59.108 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T06:59:59.108 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T06:59:59.109 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T06:59:59.109 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T06:59:59.109 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:59.109 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T06:59:59.109 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T06:59:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T06:59:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T06:59:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T06:59:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T06:59:59.110 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T06:59:59.111 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T06:59:59.111 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T06:59:59.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T06:59:59.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T06:59:59.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T06:59:59.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T06:59:59.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T06:59:59.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T06:59:59.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T06:59:59.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T06:59:59.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T06:59:59.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T06:59:59.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T06:59:59.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T06:59:59.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:00.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:00.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:00.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:00.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:00.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:00.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:00.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:00.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:00.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:00.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976329 2021-07-26T07:00:00.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976329 '!=' null 2021-07-26T07:00:00.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:00.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:00.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:00.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:00.627 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:00.627 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:00.628 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:00.628 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:00.629 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:00.629 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:00.629 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:00.629 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:00.629 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:00.630 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:00.630 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:00.630 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:00.630 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:00.631 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:00.631 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:00.631 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:00.631 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:00.631 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:00.632 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:00.632 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:00.632 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:00.632 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:00.633 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:00.633 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:00.633 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:00.633 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:00.634 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:00.634 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:00.634 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:00.634 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:00.634 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:00.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:00.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:00.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:00.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:00.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:00.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:00.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:00.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:00.740 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:01.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:01.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:01.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:01.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:01.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:01.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:01.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:01.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:01.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:01.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:01.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:01.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:01.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:02.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091066 2021-07-26T07:00:02.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091066 '!=' null 2021-07-26T07:00:02.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:02.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:02.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:02.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:02.248 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:02.249 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:02.250 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:02.250 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:02.250 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:02.251 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:02.251 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:02.252 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:02.253 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:02.253 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:02.254 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:02.254 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:02.254 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:02.254 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:02.255 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:02.255 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:02.255 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:02.255 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:02.255 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:02.256 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:02.256 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:02.256 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:02.256 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:02.256 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:02.257 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:02.257 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:02.257 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:02.257 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:02.257 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:02.258 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:02.258 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:02.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:02.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:02.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:02.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:02.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:02.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:02.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:02.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:02.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:02.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:02.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:02.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:02.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:03.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:03.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:03.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:03.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:03.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:03.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:03.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:03.150 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:03.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:03.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976609 2021-07-26T07:00:03.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976609 '!=' null 2021-07-26T07:00:03.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:03.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:03.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:03.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:03.861 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:03.861 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:03.861 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:03.862 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:03.862 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:03.862 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:03.862 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:03.863 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:03.863 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:03.863 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:03.863 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:03.864 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:03.864 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:03.864 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:03.865 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:03.865 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:03.865 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:03.865 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:03.865 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:03.866 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:03.866 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:03.866 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:03.866 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:03.867 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:03.867 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:03.867 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:03.867 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:03.867 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:03.868 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:03.868 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:03.868 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:03.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:03.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:03.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:03.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:03.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:03.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:03.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:03.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:03.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:04.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:04.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:04.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:04.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:05.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:05.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:05.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:05.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:05.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:05.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:05.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:05.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:05.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:05.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976610 2021-07-26T07:00:05.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976610 '!=' null 2021-07-26T07:00:05.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:05.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:05.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:05.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:05.638 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:05.639 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:05.639 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:05.639 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:05.640 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:05.640 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:05.640 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:05.640 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:05.640 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:05.641 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:05.641 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:05.641 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:05.641 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:05.642 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:05.642 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:05.642 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:05.642 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:05.642 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:05.643 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:05.643 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:05.643 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:05.643 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:05.644 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:05.644 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:05.644 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:05.644 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:05.644 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:05.645 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:05.645 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:05.645 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:05.645 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:05.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:05.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:05.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:05.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:05.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:05.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:05.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:05.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:05.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:06.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:06.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:06.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:06.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:06.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:06.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:06.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:06.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:06.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:06.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:06.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:06.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:06.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:07.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976610 2021-07-26T07:00:07.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976610 '!=' null 2021-07-26T07:00:07.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:07.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:07.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:07.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:07.454 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:07.455 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:07.455 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:07.455 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:07.455 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:07.455 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:07.456 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:07.456 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:07.456 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:07.456 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:07.457 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:07.457 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:07.457 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:07.457 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:07.457 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:07.458 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:07.458 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:07.458 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:07.458 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:07.458 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:07.459 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:07.459 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:07.459 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:07.459 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:07.460 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:07.460 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:07.460 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:07.460 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:07.460 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:07.461 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:07.461 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:07.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:07.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:07.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:07.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:07.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:07.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:07.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:07.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:07.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:07.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:07.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:07.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:07.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:08.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:08.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:08.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:08.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:08.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:08.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:08.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091538 2021-07-26T07:00:08.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091538 '!=' null 2021-07-26T07:00:08.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:08.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:08.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:08.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:09.062 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:09.063 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:09.064 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:09.064 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:09.065 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:09.065 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:09.065 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:09.066 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:09.066 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:09.067 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:09.067 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:09.067 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:09.067 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:09.068 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:09.068 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:09.069 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:09.069 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:09.069 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:09.070 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:09.071 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:09.071 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:09.071 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:09.072 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:09.072 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:09.072 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:09.073 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:09.073 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:09.073 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:09.073 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:09.074 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:09.074 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:09.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:09.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:09.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:09.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:09.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:09.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:09.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:09.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:09.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:09.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:09.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:09.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:09.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:10.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:10.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:10.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:10.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:10.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:10.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:10.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:10.028 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:10.028 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:10.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229699 2021-07-26T07:00:10.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229699 '!=' null 2021-07-26T07:00:10.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:10.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:10.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:10.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:10.584 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:10.585 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:10.585 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:10.585 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:10.585 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:10.586 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:10.586 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:10.586 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:10.587 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:10.587 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:10.587 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:10.587 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:10.588 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:10.588 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:10.589 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:10.589 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:10.589 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:10.589 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:10.590 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:10.591 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:10.591 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:10.592 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:10.592 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:10.592 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:10.592 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:10.593 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:10.593 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:10.593 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:10.594 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:10.594 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:10.594 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:10.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:10.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:10.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:10.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:10.697 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:10.697 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:10.698 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:10.698 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:10.699 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:11.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:11.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:11.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:11.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:11.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:11.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:11.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:11.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:11.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:11.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:11.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:11.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:11.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:12.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218130 2021-07-26T07:00:12.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218130 '!=' null 2021-07-26T07:00:12.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:12.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:12.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:12.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:12.218 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:12.219 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:12.219 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:12.220 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:12.220 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:12.220 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:12.221 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:12.221 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:12.222 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:12.223 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:12.223 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:12.223 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:12.224 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:12.224 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:12.225 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:12.225 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:12.225 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:12.226 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:12.226 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:12.227 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:12.227 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:12.227 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:12.228 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:12.228 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:12.229 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:12.229 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:12.229 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:12.230 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:12.230 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:12.230 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:12.231 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:12.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:12.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:12.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:12.330 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:12.330 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:12.330 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:12.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:12.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:12.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:12.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:12.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:12.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:12.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:13.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:13.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:13.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:13.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:13.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:13.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:13.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:13.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:13.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:13.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103235 2021-07-26T07:00:13.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103235 '!=' null 2021-07-26T07:00:13.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:13.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:13.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:13.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:13.882 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:13.882 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:13.883 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:13.883 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:13.883 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:13.883 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:13.884 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:13.884 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:13.884 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:13.884 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:13.884 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:13.885 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:13.885 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:13.885 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:13.886 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:13.886 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:13.886 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:13.886 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:13.887 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:13.887 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:13.887 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:13.887 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:13.887 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:13.888 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:13.888 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:13.888 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:13.889 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:13.889 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:13.889 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:13.889 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:13.889 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:13.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:13.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:13.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:13.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:13.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:13.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:13.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:13.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:13.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:14.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:14.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:14.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:14.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:14.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:14.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:14.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:14.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:14.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:14.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:14.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:14.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:14.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:15.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103235 2021-07-26T07:00:15.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103235 '!=' null 2021-07-26T07:00:15.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:15.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:15.364 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:15.364 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:15.446 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:15.446 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:15.446 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:15.447 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:15.447 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:15.447 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:15.447 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:15.448 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:15.448 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:15.448 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:15.448 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:15.449 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:15.449 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:15.449 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:15.449 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:15.449 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:15.450 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:15.450 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:15.450 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:15.450 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:15.451 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:15.451 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:15.451 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:15.451 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:15.452 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:15.452 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:15.452 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:15.452 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:15.452 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:15.453 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:15.453 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:15.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:15.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:15.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:15.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:15.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:15.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:15.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:15.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:15.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:15.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:15.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:15.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:15.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:16.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:16.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:16.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:16.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:16.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:16.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:16.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:16.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:16.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:17.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102939 2021-07-26T07:00:17.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102939 '!=' null 2021-07-26T07:00:17.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:17.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:17.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:17.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:17.187 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:17.188 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:17.188 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:17.188 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:17.189 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:17.189 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:17.189 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:17.189 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:17.190 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:17.190 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:17.190 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:17.190 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:17.191 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:17.191 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:17.191 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:17.191 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:17.191 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:17.192 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:17.192 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:17.192 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:17.192 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:17.193 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:17.193 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:17.193 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:17.193 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:17.193 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:17.194 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:17.194 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:17.194 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:17.195 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:17.195 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:17.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:17.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:17.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:17.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:17.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:17.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:17.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:17.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:17.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:17.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:17.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:17.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:17.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:18.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:18.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:18.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:18.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:18.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:18.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:18.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:18.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:18.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:18.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20091384 2021-07-26T07:00:18.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20091384 '!=' null 2021-07-26T07:00:18.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:18.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:18.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:18.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:18.705 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:18.705 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:18.705 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:18.706 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:18.706 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:18.706 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:18.706 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:18.707 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:18.707 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:18.707 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:18.707 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:18.707 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:18.708 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:18.708 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:18.708 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:18.708 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:18.708 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:18.709 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:18.709 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:18.709 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:18.709 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:18.710 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:18.710 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:18.710 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:18.710 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:18.710 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:18.711 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:18.711 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:18.711 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:18.711 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:18.711 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:18.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:18.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:18.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:18.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:18.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:18.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:18.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:18.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:18.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:19.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:19.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:19.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:19.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:20.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:20.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:20.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:20.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:20.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:20.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:20.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:20.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:20.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:20.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976500 2021-07-26T07:00:20.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976500 '!=' null 2021-07-26T07:00:20.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:20.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:20.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:20.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:20.572 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:20.572 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:20.573 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:20.573 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:20.573 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:20.573 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:20.573 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:20.574 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:20.574 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:20.574 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:20.574 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:20.574 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:20.575 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:20.575 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:20.575 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:20.575 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:20.576 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:20.576 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:20.576 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:20.576 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:20.576 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:20.577 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:20.577 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:20.577 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:20.577 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:20.577 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:20.578 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:20.578 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:20.578 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:20.578 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:20.578 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:20.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:20.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:20.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:20.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:20.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:20.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:20.686 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:20.686 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:20.687 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:21.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:21.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:21.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:21.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:21.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:21.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:21.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:21.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:21.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:21.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:21.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:21.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:21.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:21.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20965023 2021-07-26T07:00:21.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20965023 '!=' null 2021-07-26T07:00:21.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:21.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:21.968 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:21.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:22.049 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:22.050 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:22.051 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:22.051 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:22.052 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:22.053 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:22.053 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:22.053 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:22.053 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:22.053 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:22.054 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:22.055 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:22.056 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:22.057 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:22.057 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:22.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:22.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:22.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:22.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:22.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:22.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:22.159 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:22.160 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:22.160 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:22.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:22.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:22.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:22.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:23.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:23.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:23.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:23.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:23.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:23.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:23.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:23.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:23.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:23.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976820 2021-07-26T07:00:23.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976820 '!=' null 2021-07-26T07:00:23.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:23.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:23.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:23.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:23.927 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:23.927 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:23.928 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:23.928 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:23.928 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:23.928 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:23.929 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:23.929 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:23.929 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:23.929 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:23.929 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:23.930 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:23.930 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:23.930 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:23.930 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:23.931 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:23.931 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:23.931 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:23.931 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:23.932 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:23.932 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:23.932 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:23.932 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:23.932 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:23.933 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:23.933 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:23.933 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:23.933 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:23.933 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:23.934 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:23.934 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:23.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:24.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:24.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:24.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:24.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:24.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:24.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:24.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:24.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:24.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:24.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:24.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:24.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:24.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:24.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:24.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:24.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:24.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:24.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:24.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:24.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:24.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:25.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976820 2021-07-26T07:00:25.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976820 '!=' null 2021-07-26T07:00:25.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:25.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:25.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:25.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:25.610 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:25.611 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:25.611 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:25.612 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:25.612 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:25.613 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:25.613 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:25.614 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:25.614 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:25.614 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:25.615 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:25.616 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:25.616 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:25.617 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:25.619 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:25.619 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:25.619 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:25.619 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:25.619 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:25.620 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:25.620 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:25.620 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:25.620 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:25.621 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:25.621 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:25.621 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:25.621 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:25.622 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:25.622 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:25.622 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:25.622 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:25.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:25.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:25.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:25.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:25.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:25.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:25.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:25.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:25.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:26.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:26.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:26.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:26.101 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:26.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:26.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:26.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:26.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:26.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:26.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:26.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:26.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:26.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:27.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976876 2021-07-26T07:00:27.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976876 '!=' null 2021-07-26T07:00:27.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:27.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:27.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:27.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:27.127 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:27.127 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:27.128 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:27.129 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:27.129 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:27.130 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:27.130 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:27.131 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:27.132 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:27.132 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:27.132 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:27.132 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:27.133 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:27.133 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:27.133 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:27.133 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:27.133 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:27.134 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:27.134 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:27.134 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:27.134 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:27.135 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:27.135 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:27.135 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:27.135 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:27.135 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:27.136 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:27.136 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:27.136 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:27.136 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:27.137 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:27.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:27.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:27.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:27.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:27.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:27.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:27.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:27.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:27.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:27.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:27.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:27.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:27.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:28.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:28.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:28.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:28.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:28.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:28.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:28.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:28.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:28.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:28.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218735 2021-07-26T07:00:28.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218735 '!=' null 2021-07-26T07:00:28.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:28.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:28.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:28.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:28.793 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:28.794 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:28.795 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:28.796 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:28.796 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:28.796 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:28.796 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:28.797 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:28.797 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:28.797 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:28.797 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:28.798 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:28.798 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:28.799 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:28.799 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:28.799 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:28.799 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:28.799 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:28.800 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:28.800 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:28.800 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:28.800 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:28.801 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:28.801 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:28.801 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:28.801 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:28.801 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:28.802 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:28.802 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:28.802 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:28.802 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:28.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:28.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:28.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:28.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:28.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:28.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:28.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:28.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:28.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:29.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:29.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:29.281 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:29.281 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:29.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:29.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:29.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:29.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:29.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:29.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:29.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:29.827 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:29.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:30.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229872 2021-07-26T07:00:30.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229872 '!=' null 2021-07-26T07:00:30.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:30.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:30.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:30.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:30.453 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:30.453 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:30.454 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:30.454 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:30.454 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:30.454 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:30.455 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:30.455 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:30.456 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:30.456 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:30.457 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:30.457 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:30.458 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:30.458 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:30.459 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:30.459 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:30.460 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:30.460 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:30.460 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:30.461 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:30.462 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:30.462 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:30.463 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:30.463 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:30.463 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:30.463 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:30.464 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:30.464 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:30.464 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:30.464 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:30.465 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:30.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:30.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:30.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:30.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:30.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:30.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:30.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:30.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:30.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:30.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:30.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:30.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:30.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:31.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:31.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:31.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:31.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:31.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:31.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:31.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:31.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:31.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:32.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344862 2021-07-26T07:00:32.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344862 '!=' null 2021-07-26T07:00:32.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:32.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:32.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:32.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:32.323 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:32.323 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:32.324 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:32.324 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:32.325 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:32.325 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:32.326 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:32.326 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:32.326 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:32.327 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:32.327 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:32.328 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:32.328 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:32.329 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:32.329 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:32.329 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:32.330 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:32.330 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:32.331 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:32.331 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:32.332 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:32.332 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:32.332 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:32.333 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:32.333 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:32.333 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:32.334 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:32.334 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:32.334 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:32.334 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:32.335 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:32.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:32.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:32.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:32.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:32.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:32.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:32.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:32.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:32.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:32.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:32.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:32.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:32.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:33.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:33.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:33.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:33.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:33.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:33.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:33.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:33.282 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:33.282 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:33.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356144 2021-07-26T07:00:33.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356144 '!=' null 2021-07-26T07:00:33.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:33.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:33.829 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:33.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:33.908 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:33.909 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:33.909 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:33.909 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:33.910 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:33.910 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:33.910 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:33.911 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:33.911 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:33.911 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:33.911 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:33.912 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:33.912 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:33.912 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:33.912 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:33.912 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:33.913 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:33.913 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:33.913 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:33.913 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:33.914 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:33.914 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:33.914 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:33.914 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:33.914 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:33.915 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:33.915 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:33.915 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:33.915 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:33.915 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:33.916 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:33.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:34.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:34.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:34.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:34.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:34.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:34.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:34.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:34.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:34.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:34.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:34.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:34.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:34.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:34.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:34.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:34.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:34.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:34.826 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:35.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355868 2021-07-26T07:00:35.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355868 '!=' null 2021-07-26T07:00:35.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:35.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:35.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:35.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:35.430 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:35.430 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:35.431 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:35.431 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:35.431 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:35.432 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:35.432 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:35.432 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:35.432 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:35.433 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:35.433 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:35.434 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:35.434 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:35.434 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:35.434 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:35.434 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:35.435 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:35.435 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:35.435 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:35.435 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:35.436 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:35.436 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:35.436 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:35.436 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:35.437 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:35.437 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:35.437 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:35.437 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:35.438 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:35.438 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:35.438 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:35.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:35.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:35.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:35.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:35.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:35.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:35.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:35.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:35.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:35.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:35.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:35.939 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:35.939 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:36.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:36.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:36.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:36.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:36.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:36.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:36.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:36.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:36.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:36.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355868 2021-07-26T07:00:36.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355868 '!=' null 2021-07-26T07:00:36.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:36.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:36.920 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:36.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:37.002 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:37.003 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:37.003 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:37.003 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:37.004 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:37.004 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:37.004 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:37.004 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:37.005 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:37.005 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:37.005 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:37.005 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:37.006 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:37.006 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:37.006 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:37.006 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:37.006 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:37.007 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:37.007 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:37.007 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:37.007 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:37.008 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:37.008 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:37.008 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:37.008 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:37.009 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:37.009 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:37.009 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:37.009 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:37.010 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:37.010 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:37.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:37.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:37.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:37.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:37.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:37.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:37.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:37.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:37.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:37.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:37.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:37.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:37.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:38.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:38.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:38.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:38.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:38.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:38.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:38.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:38.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:38.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:38.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344112 2021-07-26T07:00:38.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344112 '!=' null 2021-07-26T07:00:38.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:38.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:38.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:38.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:38.864 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:38.865 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:38.865 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:38.865 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:38.865 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:38.866 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:38.866 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:38.866 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:38.866 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:38.867 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:38.867 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:38.867 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:38.867 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:38.868 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:38.868 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:38.868 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:38.868 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:38.869 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:38.869 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:38.869 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:38.869 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:38.870 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:38.870 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:38.870 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:38.870 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:38.871 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:38.871 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:38.871 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:38.872 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:38.872 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:38.872 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:38.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:38.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:38.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:38.974 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:38.974 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:38.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:38.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:38.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:38.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:39.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:39.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:39.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:39.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:39.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:39.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:39.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:39.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:39.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:39.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:39.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:39.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:39.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:40.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102833 2021-07-26T07:00:40.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102833 '!=' null 2021-07-26T07:00:40.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:40.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:40.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:40.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:40.495 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:40.495 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:40.496 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:40.496 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:40.496 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:40.496 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:40.497 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:40.497 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:40.497 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:40.497 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:40.497 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:40.498 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:40.498 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:40.499 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:40.499 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:40.499 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:40.499 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:40.500 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:40.500 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:40.500 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:40.500 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:40.501 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:40.501 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:40.501 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:40.501 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:40.502 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:40.502 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:40.502 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:40.502 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:40.503 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:40.503 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:40.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:40.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:40.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:40.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:40.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:40.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:40.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:40.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:40.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:41.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:41.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:41.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:41.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:41.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:41.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:41.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:41.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:41.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:41.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:41.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:41.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:41.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:42.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470668 2021-07-26T07:00:42.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470668 '!=' null 2021-07-26T07:00:42.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:42.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:42.182 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:42.182 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:42.264 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:42.264 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:42.265 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:42.265 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:42.265 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:42.265 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:42.265 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:42.266 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:42.266 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:42.266 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:42.266 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:42.267 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:42.267 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:42.267 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:42.267 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:42.267 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:42.268 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:42.268 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:42.268 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:42.268 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:42.269 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:42.269 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:42.269 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:42.269 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:42.269 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:42.270 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:42.270 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:42.270 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:42.270 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:42.271 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:42.271 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:42.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:42.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:42.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:42.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:42.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:42.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:42.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:42.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:42.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:42.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:42.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:42.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:42.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:43.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:43.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:43.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:43.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:43.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:43.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:43.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:43.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:43.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:43.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355913 2021-07-26T07:00:43.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355913 '!=' null 2021-07-26T07:00:43.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:43.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:43.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:43.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:43.994 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:43.994 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:43.995 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:43.996 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:43.997 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:43.998 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:43.998 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:43.998 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:43.999 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:43.999 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:43.999 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:43.999 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:44.000 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:44.000 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:44.000 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:44.000 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:44.001 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:44.001 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:44.001 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:44.001 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:44.001 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:44.002 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:44.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:44.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:44.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:44.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:44.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:44.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:44.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:44.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:44.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:44.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:44.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:44.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:44.527 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:44.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:44.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:44.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:44.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:44.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:44.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:44.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:44.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:44.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:45.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355685 2021-07-26T07:00:45.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355685 '!=' null 2021-07-26T07:00:45.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:45.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:45.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:45.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:45.786 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:45.787 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:45.787 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:45.787 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:45.787 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:45.787 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:45.788 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:45.788 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:45.788 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:45.788 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:45.789 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:45.789 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:45.789 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:45.789 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:45.789 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:45.790 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:45.790 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:45.790 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:45.790 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:45.790 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:45.791 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:45.791 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:45.791 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:45.791 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:45.792 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:45.792 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:45.792 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:45.792 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:45.792 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:45.793 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:45.793 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:45.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:45.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:45.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:45.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:45.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:45.898 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:45.898 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:45.898 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:45.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:46.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:46.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:46.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:46.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:46.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:46.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:46.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:46.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:46.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:47.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355685 2021-07-26T07:00:47.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355685 '!=' null 2021-07-26T07:00:47.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:47.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:47.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:47.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:47.399 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:47.399 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:47.400 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:47.400 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:47.400 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:47.400 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:47.401 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:47.401 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:47.401 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:47.401 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:47.402 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:47.402 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:47.402 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:47.402 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:47.402 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:47.403 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:47.403 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:47.403 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:47.403 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:47.404 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:47.404 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:47.404 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:47.404 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:47.404 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:47.405 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:47.405 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:47.405 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:47.405 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:47.405 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:47.406 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:47.406 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:47.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:47.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:47.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:47.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:47.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:47.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:47.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:47.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:47.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:47.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:47.886 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:47.886 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:47.886 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:48.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:48.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:48.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:48.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:48.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:48.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:48.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:48.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:48.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:48.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16596928 2021-07-26T07:00:48.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16596928 '!=' null 2021-07-26T07:00:48.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:48.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:48.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:48.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:49.065 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:49.065 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:49.065 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:49.066 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:49.066 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:49.066 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:49.066 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:49.066 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:49.067 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:49.067 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:49.067 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:49.067 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:49.068 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:49.068 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:49.068 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:49.068 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:49.068 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:49.069 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:49.069 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:49.069 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:49.069 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:49.070 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:49.070 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:49.070 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:49.070 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:49.070 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:49.071 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:49.071 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:49.071 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:49.071 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:49.072 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:49.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:49.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:49.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:49.174 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:49.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:49.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:49.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:49.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:49.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:49.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:49.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:49.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:49.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:50.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:50.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:50.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:50.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:50.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:50.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:50.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:50.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:50.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=9608737 2021-07-26T07:00:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 9608737 '!=' null 2021-07-26T07:00:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:50.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:50.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:50.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:51.005 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:51.005 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:51.005 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:51.006 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:51.006 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:51.006 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:51.006 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:51.006 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:51.007 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:51.007 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:51.007 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:51.007 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:51.008 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:51.008 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:51.008 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:51.008 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:51.008 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:51.009 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:51.009 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:51.009 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:51.009 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:51.010 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:51.010 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:51.010 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:51.010 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:51.010 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:51.011 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:51.011 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:51.011 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:51.011 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:51.012 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:51.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:51.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:51.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:51.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:51.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:51.115 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:51.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:51.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:51.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:51.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:51.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:51.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:51.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:52.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:52.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:52.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:52.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:52.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:52.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:52.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:52.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:52.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:52.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15723195 2021-07-26T07:00:52.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15723195 '!=' null 2021-07-26T07:00:52.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:52.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:52.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:52.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:52.608 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:52.608 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:52.608 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:52.609 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:52.609 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:52.609 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:52.609 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:52.609 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:52.610 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:52.610 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:52.610 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:52.610 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:52.611 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:52.611 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:52.611 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:52.611 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:52.612 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:52.612 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:52.612 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:52.612 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:52.612 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:52.613 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:52.613 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:52.613 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:52.613 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:52.614 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:52.614 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:52.614 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:52.614 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:52.615 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:52.615 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:52.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:52.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:52.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:52.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:52.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:52.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:52.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:52.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:52.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:53.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:53.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:53.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:53.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:53.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:53.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:53.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:53.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:53.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:53.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:53.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:53.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:53.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:53.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355602 2021-07-26T07:00:53.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355602 '!=' null 2021-07-26T07:00:53.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:53.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:53.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:53.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:54.011 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:54.012 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:54.012 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:54.012 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:54.012 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:54.013 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:54.013 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:54.013 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:54.013 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:54.014 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:54.014 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:54.014 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:54.014 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:54.014 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:54.015 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:54.015 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:54.015 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:54.015 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:54.016 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:54.016 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:54.016 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:54.016 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:54.016 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:54.017 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:54.017 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:54.017 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:54.017 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:54.017 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:54.018 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:54.018 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:54.018 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:54.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:54.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:54.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:54.127 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:54.127 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:54.127 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:54.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:54.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:54.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:54.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:54.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:54.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:54.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:54.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:54.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:54.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:54.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:54.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:54.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:54.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:54.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:54.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:55.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355640 2021-07-26T07:00:55.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355640 '!=' null 2021-07-26T07:00:55.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:55.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:55.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:55.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:55.583 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:55.584 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:55.585 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:55.586 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:55.587 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:55.587 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:55.587 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:55.587 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:55.588 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:55.588 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:55.588 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:55.588 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:55.588 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:55.589 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:55.589 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:55.589 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:55.589 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:55.589 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:55.590 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:55.590 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:55.590 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:55.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:55.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:55.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:55.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:55.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:55.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:55.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:55.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:55.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:56.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:56.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:56.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:56.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:56.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:56.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:56.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:56.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:57.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355640 2021-07-26T07:00:57.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355640 '!=' null 2021-07-26T07:00:57.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:57.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:57.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:57.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:57.207 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:57.208 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:57.208 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:57.208 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:57.209 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:57.209 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:57.209 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:57.209 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:57.210 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:57.210 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:57.210 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:57.210 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:57.211 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:57.211 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:57.211 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:57.211 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:57.211 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:57.212 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:57.212 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:57.212 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:57.213 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:57.213 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:57.213 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:57.214 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:57.214 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:57.215 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:57.215 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:57.215 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:57.215 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:57.215 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:57.216 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:57.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:57.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:57.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:57.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:57.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:57.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:57.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:57.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:57.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:57.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:57.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:57.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:57.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:58.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:58.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:58.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:58.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:58.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:58.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:58.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:58.127 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:00:58.127 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:58.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470902 2021-07-26T07:00:58.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470902 '!=' null 2021-07-26T07:00:58.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:00:58.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:58.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:00:58.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:00:58.810 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:00:58.811 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:00:58.811 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:58.811 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:58.811 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:00:58.811 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:58.812 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:00:58.812 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:00:58.812 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:58.813 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:00:58.813 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:58.813 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:58.814 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:58.814 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:58.815 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:58.815 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:00:58.815 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:00:58.815 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:00:58.816 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:00:58.816 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:00:58.816 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:00:58.816 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:58.816 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:00:58.817 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:00:58.817 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:00:58.817 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:00:58.817 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:00:58.818 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:00:58.818 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:00:58.818 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:00:58.818 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:00:58.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:00:58.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:00:58.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:00:58.920 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:00:58.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:00:58.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:00:58.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:00:58.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:00:58.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:00:59.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:00:59.281 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:00:59.281 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:00:59.282 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:00:59.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:00:59.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:00:59.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:00:59.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:00:59.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:00:59.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:00:59.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:00:59.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:00:59.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:00.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229701 2021-07-26T07:01:00.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229701 '!=' null 2021-07-26T07:01:00.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:00.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:00.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:00.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:00.361 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:00.361 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:00.361 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:00.361 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:00.362 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:00.362 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:00.362 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:00.362 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:00.363 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:00.363 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:00.363 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:00.363 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:00.363 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:00.364 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:00.364 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:00.364 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:00.364 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:00.364 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:00.365 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:00.365 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:00.365 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:00.365 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:00.366 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:00.366 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:00.366 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:00.366 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:00.366 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:00.367 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:00.367 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:00.367 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:00.367 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:00.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:00.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:00.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:00.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:00.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:00.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:00.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:00.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:00.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:00.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:00.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:00.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:00.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:01.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:01.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:01.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:01.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:01.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:01.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:01.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:01.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:01.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:02.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344489 2021-07-26T07:01:02.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344489 '!=' null 2021-07-26T07:01:02.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:02.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:02.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:02.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:02.215 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:02.215 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:02.216 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:02.216 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:02.216 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:02.216 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:02.217 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:02.217 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:02.217 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:02.217 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:02.218 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:02.218 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:02.218 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:02.218 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:02.218 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:02.219 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:02.219 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:02.219 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:02.219 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:02.219 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:02.220 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:02.220 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:02.220 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:02.220 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:02.221 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:02.221 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:02.221 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:02.221 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:02.221 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:02.222 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:02.222 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:02.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:02.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:02.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:02.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:02.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:02.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:02.328 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:02.328 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:02.328 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:02.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:02.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:02.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:02.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:03.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:03.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:03.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:03.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:03.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:03.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:03.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:03.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:03.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:03.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229525 2021-07-26T07:01:03.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229525 '!=' null 2021-07-26T07:01:03.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:03.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:03.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:03.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:03.803 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:03.803 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:03.804 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:03.804 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:03.804 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:03.804 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:03.804 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:03.805 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:03.805 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:03.805 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:03.805 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:03.805 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:03.805 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:03.806 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:03.806 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:03.806 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:03.806 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:03.806 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:03.807 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:03.807 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:03.807 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:03.807 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:03.807 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:03.808 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:03.808 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:03.808 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:03.808 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:03.808 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:03.809 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:03.809 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:03.809 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:03.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:03.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:03.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:03.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:03.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:03.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:03.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:03.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:03.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:04.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:04.267 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:04.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:04.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:04.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:04.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:04.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:04.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:04.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:04.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:04.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:04.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:04.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:05.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229525 2021-07-26T07:01:05.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229525 '!=' null 2021-07-26T07:01:05.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:05.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:05.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:05.462 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:05.544 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:05.545 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:05.545 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:05.545 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:05.545 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:05.546 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:05.546 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:05.546 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:05.547 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:05.547 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:05.547 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:05.547 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:05.548 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:05.548 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:05.548 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:05.549 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:05.549 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:05.549 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:05.549 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:05.550 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:05.550 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:05.550 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:05.550 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:05.551 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:05.551 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:05.551 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:05.551 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:05.552 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:05.552 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:05.552 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:05.553 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:05.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:05.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:05.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:05.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:05.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:05.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:05.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:05.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:05.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:06.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:06.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:06.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:06.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:06.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:06.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:06.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:06.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:06.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:06.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:06.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:06.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:06.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:06.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229552 2021-07-26T07:01:06.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229552 '!=' null 2021-07-26T07:01:06.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:06.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:06.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:06.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:07.026 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:07.027 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:07.027 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:07.027 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:07.027 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:07.028 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:07.028 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:07.028 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:07.028 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:07.029 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:07.029 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:07.029 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:07.029 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:07.029 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:07.030 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:07.030 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:07.030 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:07.030 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:07.031 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:07.031 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:07.031 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:07.031 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:07.031 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:07.032 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:07.032 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:07.032 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:07.032 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:07.032 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:07.033 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:07.033 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:07.033 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:07.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:07.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:07.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:07.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:07.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:07.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:07.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:07.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:07.139 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:07.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:07.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:07.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:07.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:08.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:08.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:08.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:08.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:08.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:08.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:08.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:08.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:08.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:08.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217995 2021-07-26T07:01:08.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217995 '!=' null 2021-07-26T07:01:08.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:08.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:08.939 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:08.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:09.024 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:09.024 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:09.024 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:09.024 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:09.024 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:09.025 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:09.025 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:09.025 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:09.025 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:09.025 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:09.026 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:09.026 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:09.026 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:09.026 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:09.026 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:09.027 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:09.027 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:09.027 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:09.027 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:09.027 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:09.028 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:09.028 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:09.028 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:09.028 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:09.029 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:09.029 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:09.029 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:09.029 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:09.029 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:09.030 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:09.030 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:09.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:09.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:09.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:09.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:09.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:09.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:09.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:09.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:09.137 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:09.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:09.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:09.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:09.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:10.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:10.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:10.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:10.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:10.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:10.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:10.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:10.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102913 2021-07-26T07:01:10.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102913 '!=' null 2021-07-26T07:01:10.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:10.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:10.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:10.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:10.652 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:10.652 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:10.653 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:10.653 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:10.653 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:10.653 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:10.654 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:10.654 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:10.654 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:10.655 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:10.655 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:10.655 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:10.656 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:10.656 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:10.656 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:10.656 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:10.657 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:10.657 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:10.657 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:10.657 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:10.658 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:10.658 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:10.658 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:10.658 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:10.658 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:10.659 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:10.659 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:10.659 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:10.659 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:10.660 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:10.660 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:10.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:10.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:10.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:10.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:10.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:10.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:10.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:10.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:10.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:11.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:11.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:11.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:11.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:11.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:11.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:11.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:11.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:11.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:11.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:11.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:11.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:11.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:12.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102926 2021-07-26T07:01:12.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102926 '!=' null 2021-07-26T07:01:12.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:12.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:12.265 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:12.265 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:12.347 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:12.348 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:12.348 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:12.348 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:12.348 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:12.349 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:12.349 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:12.349 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:12.349 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:12.350 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:12.350 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:12.350 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:12.350 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:12.350 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:12.351 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:12.351 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:12.351 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:12.351 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:12.352 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:12.352 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:12.352 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:12.352 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:12.352 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:12.353 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:12.353 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:12.353 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:12.353 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:12.353 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:12.354 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:12.354 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:12.354 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:12.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:12.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:12.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:12.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:12.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:12.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:12.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:12.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:12.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:12.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:12.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:12.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:12.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:13.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:13.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:13.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:13.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:13.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:13.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:13.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:13.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:13.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:13.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103205 2021-07-26T07:01:13.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103205 '!=' null 2021-07-26T07:01:13.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:13.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:13.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:13.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:13.994 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:13.994 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:13.994 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:13.994 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:13.995 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:13.995 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:13.995 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:13.995 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:13.996 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:13.996 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:13.996 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:13.996 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:13.996 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:13.997 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:13.997 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:13.997 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:13.997 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:13.997 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:13.998 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:13.998 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:13.999 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:13.999 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:13.999 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:13.999 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:14.000 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:14.000 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:14.000 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:14.000 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:14.001 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:14.001 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:14.001 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:14.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:14.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:14.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:14.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:14.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:14.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:14.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:14.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:14.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:14.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:14.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:14.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:14.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:15.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:15.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:15.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:15.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:15.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:15.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:15.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:15.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:15.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:15.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103385 2021-07-26T07:01:15.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103385 '!=' null 2021-07-26T07:01:15.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:15.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:15.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:15.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:15.861 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:15.861 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:15.861 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:15.862 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:15.862 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:15.862 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:15.862 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:15.862 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:15.863 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:15.863 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:15.863 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:15.863 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:15.864 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:15.864 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:15.864 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:15.864 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:15.864 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:15.865 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:15.865 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:15.865 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:15.865 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:15.866 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:15.866 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:15.866 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:15.866 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:15.866 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:15.867 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:15.867 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:15.867 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:15.867 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:15.868 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:15.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:15.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:15.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:15.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:15.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:15.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:15.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:15.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:15.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:16.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:16.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:16.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:16.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:16.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:16.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:16.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:16.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103385 2021-07-26T07:01:17.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103385 '!=' null 2021-07-26T07:01:17.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:17.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:17.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:17.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:17.464 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:17.465 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:17.466 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:17.467 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:17.468 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:17.469 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:17.470 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:17.471 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:17.472 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:17.472 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:17.472 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:17.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:17.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:17.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:17.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:17.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:17.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:17.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:17.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:17.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:17.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:17.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:17.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:17.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:18.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:18.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:18.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:18.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:18.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:18.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:18.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:18.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:18.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:18.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345236 2021-07-26T07:01:18.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345236 '!=' null 2021-07-26T07:01:18.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:18.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:18.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:18.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:19.000 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:19.000 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:19.001 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:19.001 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:19.001 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:19.001 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:19.002 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:19.002 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:19.002 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:19.002 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:19.003 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:19.003 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:19.003 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:19.004 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:19.004 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:19.004 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:19.004 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:19.004 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:19.005 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:19.005 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:19.005 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:19.005 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:19.005 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:19.006 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:19.006 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:19.006 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:19.006 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:19.006 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:19.007 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:19.007 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:19.007 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:19.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:19.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:19.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:19.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:19.110 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:19.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:19.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:19.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:19.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:19.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:19.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:19.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:19.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:19.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:19.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:19.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:19.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:19.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:19.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:19.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:19.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:19.888 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:20.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356611 2021-07-26T07:01:20.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356611 '!=' null 2021-07-26T07:01:20.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:20.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:20.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:20.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:20.602 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:20.603 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:20.603 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:20.603 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:20.603 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:20.604 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:20.604 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:20.604 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:20.604 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:20.605 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:20.605 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:20.605 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:20.605 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:20.605 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:20.606 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:20.606 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:20.606 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:20.606 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:20.606 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:20.607 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:20.607 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:20.607 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:20.607 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:20.607 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:20.608 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:20.608 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:20.608 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:20.608 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:20.609 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:20.609 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:20.609 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:20.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:20.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:20.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:20.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:20.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:20.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:20.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:20.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:20.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:21.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:21.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:21.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:21.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:21.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:21.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:21.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:21.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:21.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:21.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:21.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:21.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:21.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:22.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356813 2021-07-26T07:01:22.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356813 '!=' null 2021-07-26T07:01:22.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:22.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:22.090 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:22.090 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:22.165 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:22.165 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:22.166 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:22.167 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:22.168 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:22.169 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:22.170 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:22.171 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:22.172 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:22.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:22.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:22.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:22.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:22.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:22.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:22.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:22.276 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:22.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:22.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:22.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:22.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:22.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:23.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:23.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:23.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:23.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:23.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:23.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:23.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:23.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:23.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:24.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345852 2021-07-26T07:01:24.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345852 '!=' null 2021-07-26T07:01:24.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:24.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:24.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:24.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:24.091 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:24.092 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:24.092 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:24.092 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:24.093 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:24.093 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:24.093 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:24.093 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:24.093 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:24.094 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:24.094 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:24.094 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:24.094 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:24.095 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:24.095 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:24.095 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:24.095 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:24.095 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:24.096 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:24.096 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:24.096 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:24.096 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:24.097 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:24.097 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:24.097 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:24.097 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:24.097 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:24.098 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:24.098 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:24.098 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:24.098 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:24.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:24.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:24.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:24.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:24.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:24.203 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:24.203 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:24.203 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:24.204 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:24.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:24.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:24.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:24.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:25.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:25.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:25.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:25.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:25.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:25.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:25.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:25.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:25.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:25.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12230191 2021-07-26T07:01:25.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12230191 '!=' null 2021-07-26T07:01:25.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:25.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:25.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:25.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:25.690 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:25.690 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:25.691 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:25.691 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:25.691 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:25.691 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:25.692 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:25.692 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:25.692 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:25.692 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:25.693 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:25.693 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:25.693 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:25.693 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:25.693 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:25.694 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:25.694 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:25.694 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:25.694 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:25.694 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:25.695 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:25.695 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:25.695 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:25.695 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:25.696 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:25.696 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:25.696 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:25.696 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:25.696 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:25.697 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:25.697 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:25.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:25.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:25.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:25.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:25.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:25.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:25.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:25.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:25.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:26.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:26.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:26.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:26.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:26.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:26.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:26.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:26.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:26.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:27.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345174 2021-07-26T07:01:27.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345174 '!=' null 2021-07-26T07:01:27.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:27.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:27.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:27.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:27.469 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:27.469 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:27.470 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:27.470 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:27.470 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:27.470 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:27.471 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:27.471 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:27.471 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:27.471 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:27.472 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:27.472 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:27.472 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:27.472 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:27.472 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:27.473 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:27.473 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:27.473 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:27.473 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:27.473 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:27.474 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:27.474 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:27.474 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:27.474 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:27.475 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:27.475 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:27.475 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:27.475 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:27.475 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:27.476 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:27.476 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:27.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:27.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:27.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:27.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:27.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:27.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:27.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:27.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:27.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:27.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:27.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:27.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:27.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:28.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:28.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:28.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:28.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:28.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:28.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:28.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:28.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:28.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:28.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345174 2021-07-26T07:01:28.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345174 '!=' null 2021-07-26T07:01:28.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:28.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:28.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:28.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:28.963 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:28.964 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:28.964 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:28.964 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:28.964 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:28.965 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:28.965 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:28.965 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:28.965 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:28.965 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:28.966 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:28.966 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:28.966 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:28.966 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:28.967 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:28.967 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:28.967 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:28.967 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:28.967 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:28.968 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:28.968 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:28.968 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:28.968 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:28.968 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:28.969 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:28.969 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:28.969 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:28.969 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:28.970 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:28.970 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:28.970 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:28.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:29.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:29.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:29.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:29.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:29.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:29.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:29.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:29.075 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:29.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:29.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:29.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:29.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:30.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:30.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:30.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:30.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:30.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:30.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:30.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:30.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:30.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:30.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103518 2021-07-26T07:01:30.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103518 '!=' null 2021-07-26T07:01:30.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:30.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:30.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:30.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:30.832 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:30.833 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:30.833 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:30.833 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:30.833 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:30.834 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:30.834 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:30.834 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:30.834 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:30.834 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:30.835 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:30.835 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:30.835 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:30.835 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:30.835 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:30.836 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:30.836 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:30.836 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:30.836 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:30.837 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:30.837 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:30.837 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:30.837 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:30.837 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:30.838 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:30.838 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:30.838 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:30.839 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:30.839 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:30.839 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:30.839 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:30.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:30.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:30.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:30.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:30.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:30.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:30.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:30.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:30.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:31.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:31.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:31.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:31.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:31.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:31.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:31.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:31.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:31.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:31.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:31.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:31.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:31.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:32.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103429 2021-07-26T07:01:32.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103429 '!=' null 2021-07-26T07:01:32.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:32.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:32.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:32.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:32.472 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:32.473 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:32.473 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:32.473 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:32.474 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:32.474 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:32.474 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:32.474 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:32.474 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:32.475 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:32.475 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:32.475 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:32.475 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:32.476 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:32.476 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:32.476 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:32.476 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:32.476 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:32.477 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:32.477 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:32.477 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:32.477 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:32.477 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:32.478 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:32.478 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:32.478 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:32.478 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:32.479 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:32.479 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:32.479 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:32.479 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:32.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:32.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:32.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:32.582 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:32.582 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:32.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:32.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:32.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:32.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:32.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:32.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:32.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:32.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:33.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:33.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:33.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:33.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:33.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:33.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:33.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:33.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:33.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:34.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344336 2021-07-26T07:01:34.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344336 '!=' null 2021-07-26T07:01:34.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:34.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:34.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:34.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:34.257 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:34.257 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:34.257 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:34.258 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:34.258 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:34.258 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:34.258 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:34.259 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:34.259 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:34.259 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:34.259 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:34.259 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:34.260 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:34.260 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:34.260 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:34.260 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:34.260 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:34.261 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:34.261 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:34.261 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:34.261 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:34.262 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:34.262 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:34.262 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:34.262 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:34.262 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:34.263 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:34.263 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:34.263 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:34.263 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:34.263 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:34.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:34.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:34.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:34.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:34.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:34.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:34.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:34.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:34.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:34.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:34.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:34.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:34.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:35.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:35.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:35.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:35.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:35.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:35.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:35.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:35.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:35.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:36.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355716 2021-07-26T07:01:36.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355716 '!=' null 2021-07-26T07:01:36.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:36.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:36.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:36.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:36.204 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:36.205 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:36.205 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:36.205 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:36.205 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:36.206 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:36.206 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:36.206 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:36.206 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:36.206 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:36.207 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:36.207 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:36.207 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:36.207 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:36.208 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:36.208 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:36.208 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:36.208 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:36.208 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:36.209 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:36.209 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:36.209 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:36.209 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:36.209 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:36.210 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:36.210 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:36.210 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:36.210 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:36.210 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:36.211 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:36.211 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:36.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:36.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:36.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:36.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:36.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:36.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:36.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:36.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:36.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:36.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:36.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:36.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:37.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:37.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:37.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:37.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:37.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:37.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:37.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:37.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:37.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:37.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16597075 2021-07-26T07:01:37.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16597075 '!=' null 2021-07-26T07:01:37.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:37.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:37.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:37.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:37.749 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:37.749 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:37.749 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:37.750 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:37.750 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:37.750 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:37.750 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:37.750 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:37.751 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:37.751 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:37.751 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:37.751 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:37.752 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:37.752 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:37.752 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:37.752 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:37.752 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:37.753 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:37.753 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:37.753 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:37.753 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:37.754 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:37.754 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:37.754 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:37.754 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:37.754 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:37.755 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:37.755 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:37.755 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:37.755 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:37.756 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:37.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:37.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:37.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:37.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:37.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:37.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:37.861 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:37.861 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:37.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:38.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:38.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:38.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:38.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:39.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:39.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:39.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:39.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:39.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:39.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:39.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:39.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:39.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:39.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482575 2021-07-26T07:01:39.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482575 '!=' null 2021-07-26T07:01:39.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:39.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:39.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:39.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:39.783 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:39.783 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:39.783 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:39.784 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:39.784 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:39.784 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:39.784 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:39.784 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:39.785 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:39.785 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:39.785 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:39.785 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:39.786 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:39.786 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:39.786 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:39.786 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:39.786 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:39.787 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:39.787 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:39.787 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:39.787 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:39.787 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:39.788 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:39.788 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:39.788 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:39.788 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:39.788 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:39.789 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:39.789 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:39.789 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:39.789 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:39.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:39.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:39.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:39.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:39.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:39.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:39.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:39.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:39.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:40.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:40.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:40.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:40.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:40.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:40.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:40.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:40.854 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:40.854 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:41.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482417 2021-07-26T07:01:41.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482417 '!=' null 2021-07-26T07:01:41.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:41.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:41.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:41.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:41.398 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:41.398 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:41.398 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:41.399 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:41.399 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:41.399 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:41.399 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:41.400 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:41.400 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:41.400 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:41.401 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:41.401 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:41.401 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:41.401 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:41.402 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:41.402 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:41.402 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:41.402 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:41.403 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:41.403 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:41.403 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:41.403 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:41.403 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:41.404 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:41.404 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:41.404 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:41.404 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:41.405 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:41.405 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:41.405 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:41.405 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:41.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:41.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:41.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:41.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:41.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:41.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:41.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:41.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:41.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:42.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:42.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:42.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:42.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:42.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:42.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:42.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:42.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:42.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:42.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:42.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:42.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:42.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:43.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482417 2021-07-26T07:01:43.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482417 '!=' null 2021-07-26T07:01:43.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:43.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:43.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:43.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:43.213 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:43.213 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:43.213 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:43.214 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:43.214 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:43.214 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:43.214 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:43.215 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:43.215 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:43.215 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:43.215 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:43.215 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:43.216 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:43.216 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:43.216 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:43.216 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:43.217 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:43.217 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:43.217 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:43.217 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:43.217 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:43.218 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:43.218 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:43.218 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:43.218 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:43.218 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:43.219 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:43.219 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:43.219 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:43.219 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:43.219 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:43.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:43.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:43.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:43.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:43.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:43.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:43.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:43.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:43.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:43.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:43.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:43.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:43.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:44.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:44.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:44.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:44.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:44.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:44.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:44.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:44.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:44.198 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:44.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344179 2021-07-26T07:01:45.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344179 '!=' null 2021-07-26T07:01:45.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:45.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:45.000 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:45.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:45.081 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:45.081 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:45.081 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:45.082 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:45.082 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:45.082 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:45.082 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:45.083 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:45.083 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:45.083 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:45.083 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:45.084 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:45.084 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:45.084 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:45.084 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:45.084 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:45.085 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:45.085 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:45.085 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:45.085 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:45.085 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:45.086 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:45.086 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:45.086 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:45.087 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:45.087 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:45.087 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:45.087 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:45.087 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:45.088 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:45.088 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:45.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:45.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:45.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:45.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:45.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:45.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:45.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:45.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:45.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:45.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:45.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:45.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:45.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:46.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:46.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:46.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:46.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:46.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:46.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:46.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:46.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:46.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:46.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102990 2021-07-26T07:01:46.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102990 '!=' null 2021-07-26T07:01:46.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:46.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:46.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:46.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:46.600 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:46.601 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:46.601 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:46.601 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:46.601 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:46.602 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:46.602 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:46.602 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:46.602 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:46.602 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:46.603 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:46.603 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:46.603 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:46.603 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:46.603 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:46.604 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:46.604 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:46.604 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:46.604 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:46.605 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:46.605 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:46.605 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:46.605 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:46.605 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:46.606 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:46.606 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:46.606 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:46.606 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:46.606 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:46.607 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:46.607 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:46.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:46.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:46.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:46.712 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:46.712 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:46.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:46.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:46.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:46.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:47.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:47.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:47.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:47.074 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:47.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:47.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:47.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:47.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:47.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:47.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:47.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:47.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:47.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:48.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344273 2021-07-26T07:01:48.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344273 '!=' null 2021-07-26T07:01:48.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:48.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:48.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:48.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:48.202 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:48.202 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:48.203 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:48.203 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:48.203 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:48.203 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:48.203 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:48.204 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:48.204 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:48.204 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:48.204 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:48.204 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:48.205 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:48.205 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:48.205 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:48.205 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:48.205 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:48.206 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:48.206 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:48.206 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:48.206 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:48.206 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:48.207 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:48.207 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:48.207 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:48.207 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:48.207 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:48.208 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:48.208 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:48.208 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:48.208 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:48.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:48.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:48.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:48.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:48.312 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:48.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:48.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:48.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:48.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:48.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:48.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:48.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:48.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:49.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:49.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:49.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:49.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:49.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:49.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:49.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:49.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:49.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:49.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103080 2021-07-26T07:01:49.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103080 '!=' null 2021-07-26T07:01:49.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:49.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:49.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:49.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:49.899 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:49.900 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:49.901 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:49.902 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:49.903 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:49.904 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:49.905 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:49.906 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:49.907 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:49.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:50.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:50.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:50.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:50.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:50.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:50.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:50.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:50.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:50.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:50.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:50.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:50.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:50.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:50.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:50.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:50.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:50.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:50.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:50.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:50.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:50.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:51.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103201 2021-07-26T07:01:51.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103201 '!=' null 2021-07-26T07:01:51.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:51.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:51.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:51.401 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:51.478 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:51.479 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:51.479 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:51.479 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:51.479 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:51.480 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:51.480 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:51.480 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:51.480 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:51.480 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:51.481 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:51.481 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:51.481 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:51.481 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:51.482 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:51.482 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:51.482 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:51.482 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:51.482 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:51.483 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:51.483 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:51.483 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:51.483 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:51.484 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:51.484 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:51.484 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:51.486 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:51.486 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:51.486 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:51.486 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:51.487 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:51.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:51.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:51.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:51.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:51.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:51.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:51.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:51.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:51.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:52.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:52.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:52.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:52.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:52.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:52.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:52.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:52.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:52.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:52.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:52.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:52.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:52.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:53.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103201 2021-07-26T07:01:53.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103201 '!=' null 2021-07-26T07:01:53.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:53.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:53.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:53.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:53.148 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:53.148 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:53.148 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:53.148 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:53.149 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:53.149 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:53.149 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:53.149 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:53.149 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:53.150 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:53.150 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:53.150 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:53.150 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:53.151 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:53.151 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:53.151 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:53.151 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:53.151 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:53.152 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:53.152 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:53.152 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:53.152 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:53.152 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:53.153 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:53.153 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:53.153 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:53.153 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:53.153 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:53.154 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:53.154 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:53.154 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:53.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:53.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:53.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:53.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:53.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:53.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:53.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:53.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:53.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:53.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:53.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:53.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:53.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:54.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:54.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:54.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:54.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:54.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:54.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:54.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:54.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:54.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:54.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344722 2021-07-26T07:01:54.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344722 '!=' null 2021-07-26T07:01:54.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:54.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:54.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:54.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:54.861 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:54.861 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:54.862 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:54.862 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:54.862 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [ 2021-07-26T07:01:54.863 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:54.863 INFO:tasks.workunit.client.0.smithi094.stdout: "priority": 190, 2021-07-26T07:01:54.864 INFO:tasks.workunit.client.0.smithi094.stdout: "items": [ 2021-07-26T07:01:54.864 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:54.865 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:54.865 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:54.865 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:54.865 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:54.866 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:54.866 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:54.866 INFO:tasks.workunit.client.0.smithi094.stdout: ], 2021-07-26T07:01:54.866 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:54.867 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:54.867 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "1.0", 2021-07-26T07:01:54.867 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:54.867 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:54.867 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:54.868 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:54.868 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:54.868 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:54.868 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:54.869 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:54.869 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:54.869 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:54.869 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:54.869 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:54.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:54.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:54.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:54.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:54.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:54.971 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:54.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:54.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:54.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:55.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:55.330 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:55.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:55.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:55.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:55.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:55.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:55.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:55.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:55.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:55.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:55.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:55.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:56.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482619 2021-07-26T07:01:56.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482619 '!=' null 2021-07-26T07:01:56.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:56.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:56.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:56.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:56.445 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:56.446 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:56.446 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:56.446 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:56.447 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:56.447 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:56.447 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:56.447 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:56.447 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:56.448 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:56.448 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:56.448 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:56.448 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:56.449 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:56.449 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:56.449 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:56.449 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:56.449 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:56.450 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:56.450 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:56.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:56.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:56.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:56.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:56.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:56.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:56.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:56.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:56.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:57.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:57.273 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:57.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:57.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:01:57.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:01:57.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:01:57.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:01:57.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:01:57.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:01:57.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:01:57.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:01:57.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:01:57.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:01:58.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17034858 2021-07-26T07:01:58.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17034858 '!=' null 2021-07-26T07:01:58.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:01:58.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:58.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:01:58.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:01:58.763 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:01:58.764 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:01:58.764 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:58.765 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:58.765 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:58.765 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:01:58.765 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:01:58.765 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:01:58.766 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:01:58.766 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:01:58.766 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:58.766 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:01:58.767 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:01:58.767 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:01:58.767 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:01:58.767 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:01:58.767 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:01:58.768 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:01:58.768 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:01:58.768 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:01:58.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:01:58.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:01:58.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:01:58.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:01:58.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:01:58.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:01:58.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:01:59.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:01:59.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:01:59.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:01:59.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:00.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:02:00.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:02:00.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:00.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:00.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:00.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:00.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:00.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:00.111 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:00.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17034316 2021-07-26T07:02:00.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17034316 '!=' null 2021-07-26T07:02:00.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:00.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:00.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:00.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:00.736 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:00.737 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:00.737 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:00.738 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:00.738 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:00.739 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:00.739 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:00.740 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:00.740 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:00.741 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:00.741 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:00.742 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:00.742 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:00.743 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:00.743 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:00.744 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:00.744 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:00.745 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:00.745 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:00.746 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:00.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:00.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:00.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:00.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:00.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:00.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:00.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:00.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:00.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:01.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:02:01.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:01.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:01.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:01.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:02:01.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:02:01.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:01.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:01.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:01.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:01.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:01.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:01.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:02.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=9172399 2021-07-26T07:02:02.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 9172399 '!=' null 2021-07-26T07:02:02.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:02.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:02.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:02.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:02.654 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:02.654 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:02.654 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:02.655 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:02.655 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:02.655 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:02.655 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:02.656 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:02.656 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:02.656 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:02.656 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:02.656 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:02.657 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:02.657 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:02.657 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:02.657 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:02.658 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:02.658 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:02.658 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:02.658 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:02.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:02.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:02.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:02.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:02.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:02.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:02.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:02.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:02.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:03.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:02:03.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:03.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:03.123 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:03.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 3 2021-07-26T07:02:03.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 1 '!=' 1 2021-07-26T07:02:03.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:03.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:03.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:03.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:03.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:03.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:03.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:04.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=15724786 2021-07-26T07:02:04.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 15724786 '!=' null 2021-07-26T07:02:04.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:04.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:04.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:04.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:04.377 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:04.378 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:04.378 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:04.378 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:04.378 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:04.379 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:04.379 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:04.379 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:04.379 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:04.380 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:04.380 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:04.380 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:04.380 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:04.381 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:04.381 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:04.381 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:04.381 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:04.382 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:04.382 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:04.382 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:04.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:04.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:04.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:04.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:04.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:04.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:04.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:04.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:04.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:04.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:04.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:04.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:04.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:05.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:05.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 1 2021-07-26T07:02:05.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:02:05.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=2 2021-07-26T07:02:05.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:05.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:05.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:05.387 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:05.388 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:05.388 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:05.388 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:05.388 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:05.389 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:05.389 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:05.389 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:05.389 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:05.389 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:05.390 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:05.390 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:05.390 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:05.390 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:05.391 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:05.391 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:05.391 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:05.391 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:05.392 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:05.392 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:05.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:05.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:05.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:05.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:05.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:05.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:05.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:05.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:05.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:05.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:05.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:05.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:05.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:06.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:06.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:06.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:06.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:06.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:06.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:06.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:06.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:06.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:07.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13999537 2021-07-26T07:02:07.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13999537 '!=' null 2021-07-26T07:02:07.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:07.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:07.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:07.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:07.565 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:07.565 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:07.566 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:07.567 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:07.568 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:07.569 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:07.569 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:07.569 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:07.569 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:07.570 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:07.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:07.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:07.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:07.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:07.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:07.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:07.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:07.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:07.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:08.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:08.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:08.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:08.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:09.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:09.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:09.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:09.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:09.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:09.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:09.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:09.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:09.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:09.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356648 2021-07-26T07:02:09.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356648 '!=' null 2021-07-26T07:02:09.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:09.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:09.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:09.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:09.757 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:09.757 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:09.757 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:09.758 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:09.758 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:09.758 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:09.759 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:09.759 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:09.760 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:09.761 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:09.762 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:09.762 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:09.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:09.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:09.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:09.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:09.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:09.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:09.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:09.868 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:09.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:10.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:10.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:10.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:10.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:10.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:10.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:10.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:10.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:10.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:10.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:10.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:10.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:10.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:11.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356648 2021-07-26T07:02:11.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356648 '!=' null 2021-07-26T07:02:11.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:11.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:11.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:11.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:11.316 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:11.316 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:11.317 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:11.317 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:11.317 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:11.317 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:11.318 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:11.318 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:11.318 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:11.318 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:11.318 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:11.319 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:11.319 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:11.319 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:11.319 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:11.320 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:11.320 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:11.320 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:11.320 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:11.320 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:11.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:11.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:11.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:11.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:11.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:11.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:11.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:11.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:11.428 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:11.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:11.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:11.835 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:11.835 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:12.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:12.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:12.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:12.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:12.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:12.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:12.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:12.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:12.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:12.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356687 2021-07-26T07:02:12.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356687 '!=' null 2021-07-26T07:02:12.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:12.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:12.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:12.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:12.954 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:12.954 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:12.954 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:12.954 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:12.955 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:12.955 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:12.955 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:12.955 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:12.956 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:12.956 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:12.956 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:12.956 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:12.956 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:12.957 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:12.957 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:12.957 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:12.957 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:12.958 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:12.958 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:12.958 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:12.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:13.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:13.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:13.064 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:13.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:13.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:13.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:13.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:13.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:13.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:13.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:13.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:13.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:13.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:13.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:13.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:13.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:13.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:13.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:13.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:13.836 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:13.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:14.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=16598016 2021-07-26T07:02:14.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 16598016 '!=' null 2021-07-26T07:02:14.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:14.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:14.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:14.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:14.561 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:14.561 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:14.561 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:14.561 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:14.562 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:14.562 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:14.562 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:14.562 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:14.563 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:14.563 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:14.563 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:14.563 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:14.563 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:14.564 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:14.564 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:14.564 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:14.564 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:14.565 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:14.565 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:14.565 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:14.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:14.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:14.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:14.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:14.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:14.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:14.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:14.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:14.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:15.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:15.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:15.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:15.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:16.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:16.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:16.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:16.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:16.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:16.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:16.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:16.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:16.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:16.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229968 2021-07-26T07:02:16.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229968 '!=' null 2021-07-26T07:02:16.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:16.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:16.431 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:16.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:16.510 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:16.511 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:16.511 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:16.511 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:16.512 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:16.512 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:16.512 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:16.512 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:16.512 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:16.513 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:16.513 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:16.513 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:16.513 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:16.513 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:16.514 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:16.514 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:16.514 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:16.514 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:16.515 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:16.515 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:16.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:16.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:16.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:16.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:16.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:16.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:16.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:16.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:16.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:16.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:16.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:16.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:16.973 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:17.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:17.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:17.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:17.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:17.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:17.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:17.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:17.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:17.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:17.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344802 2021-07-26T07:02:17.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344802 '!=' null 2021-07-26T07:02:17.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:17.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:17.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:17.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:17.960 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:17.961 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:17.961 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:17.961 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:17.962 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:17.962 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:17.962 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:17.962 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:17.963 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:17.963 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:17.963 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:17.963 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:17.963 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:17.964 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:17.964 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:17.964 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:17.964 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:17.965 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:17.965 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:17.965 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:17.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:18.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:18.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:18.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:18.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:18.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:18.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:18.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:18.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:18.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:18.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:18.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:18.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:19.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:19.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:19.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:19.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:19.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:19.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:19.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:19.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:19.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:19.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356361 2021-07-26T07:02:19.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356361 '!=' null 2021-07-26T07:02:19.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:19.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:19.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:19.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:19.767 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:19.768 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:19.768 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:19.768 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:19.769 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:19.769 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:19.769 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:19.770 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:19.770 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:19.771 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:19.771 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:19.772 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:19.772 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:19.772 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:19.773 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:19.773 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:19.774 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:19.774 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:19.775 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:19.775 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:19.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:19.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:19.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:19.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:19.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:19.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:19.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:19.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:19.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:20.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:20.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:20.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:20.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:20.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:20.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:20.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:20.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:20.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:20.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:20.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:20.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:20.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:21.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356361 2021-07-26T07:02:21.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356361 '!=' null 2021-07-26T07:02:21.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:21.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:21.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:21.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:21.469 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:21.469 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:21.469 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:21.469 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:21.470 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:21.470 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:21.470 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:21.470 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:21.471 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:21.472 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:21.472 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:21.472 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:21.472 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:21.473 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:21.473 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:21.473 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:21.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:21.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:21.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:21.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:21.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:21.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:21.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:21.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:21.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:21.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:21.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:21.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:21.937 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:22.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:22.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:22.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:22.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:22.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:22.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:22.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:22.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:22.470 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:22.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356573 2021-07-26T07:02:22.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356573 '!=' null 2021-07-26T07:02:22.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:22.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:22.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:22.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:23.021 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:23.022 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:23.022 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:23.022 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:23.022 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:23.023 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:23.023 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:23.023 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:23.023 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:23.024 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:23.024 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:23.024 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:23.024 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:23.024 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:23.025 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:23.025 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:23.025 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:23.025 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:23.026 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:23.026 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:23.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:23.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:23.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:23.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:23.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:23.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:23.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:23.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:23.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:23.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:23.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:23.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:23.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:24.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:24.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:24.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:24.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:24.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:24.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:24.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:24.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:24.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:25.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345191 2021-07-26T07:02:25.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345191 '!=' null 2021-07-26T07:02:25.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:25.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:25.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:25.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:25.113 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:25.114 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:25.114 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:25.114 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:25.114 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:25.114 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:25.115 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:25.115 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:25.115 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:25.115 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:25.116 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:25.116 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:25.116 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:25.116 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:25.116 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:25.117 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:25.117 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:25.117 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:25.117 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:25.117 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:25.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:25.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:25.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:25.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:25.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:25.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:25.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:25.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:25.224 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:25.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:25.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:25.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:25.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:26.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:26.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:26.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:26.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:26.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:26.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:26.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:26.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:26.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:26.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103884 2021-07-26T07:02:26.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103884 '!=' null 2021-07-26T07:02:26.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:26.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:26.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:26.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:26.793 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:26.793 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:26.794 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:26.794 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:26.794 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:26.794 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:26.795 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:26.795 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:26.795 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:26.795 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:26.796 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:26.796 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:26.796 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:26.796 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:26.796 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:26.797 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:26.797 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:26.797 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:26.797 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:26.798 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:26.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:26.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:26.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:26.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:26.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:26.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:26.905 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:26.905 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:26.905 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:27.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:27.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:27.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:27.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:27.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:27.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:27.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:27.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:27.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:27.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:27.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:27.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:27.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:28.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218373 2021-07-26T07:02:28.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218373 '!=' null 2021-07-26T07:02:28.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:28.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:28.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:28.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:28.362 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:28.363 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:28.363 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:28.363 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:28.363 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:28.364 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:28.364 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:28.364 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:28.364 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:28.365 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:28.365 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:28.365 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:28.365 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:28.366 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:28.366 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:28.366 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:28.366 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:28.367 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:28.367 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:28.367 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:28.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:28.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:28.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:28.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:28.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:28.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:28.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:28.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:28.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:28.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:28.837 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:28.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:28.838 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:29.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:29.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:29.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:29.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:29.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:29.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:29.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:29.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:29.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:29.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103424 2021-07-26T07:02:29.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103424 '!=' null 2021-07-26T07:02:29.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:29.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:29.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:29.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:30.028 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:30.028 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:30.028 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:30.029 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:30.029 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:30.029 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:30.029 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:30.029 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:30.030 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:30.030 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:30.030 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:30.030 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:30.031 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:30.031 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:30.031 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:30.031 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:30.031 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:30.032 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:30.032 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:30.032 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:30.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:30.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:30.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:30.138 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:30.139 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:30.139 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:30.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:30.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:30.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:30.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:30.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:30.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:30.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:31.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:31.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:31.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:31.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:31.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103414 2021-07-26T07:02:31.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103414 '!=' null 2021-07-26T07:02:31.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:31.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:31.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:31.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:31.932 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:31.933 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:31.933 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:31.933 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:31.933 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:31.934 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:31.934 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:31.934 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:31.934 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:31.934 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:31.935 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:31.935 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:31.935 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:31.936 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:31.936 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:31.936 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:31.936 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:31.937 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:31.937 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:31.937 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:31.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:32.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:32.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:32.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:32.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:32.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:32.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:32.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:32.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:32.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:32.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:32.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:32.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:32.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:32.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:32.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:32.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:32.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:32.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:32.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:32.861 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:32.861 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:33.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103414 2021-07-26T07:02:33.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103414 '!=' null 2021-07-26T07:02:33.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:33.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:33.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:33.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:33.651 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:33.651 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:33.652 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:33.652 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:33.652 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:33.652 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:33.653 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:33.653 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:33.653 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:33.654 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:33.654 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:33.654 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:33.655 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:33.655 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:33.655 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:33.656 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:33.656 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:33.656 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:33.657 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:33.657 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:33.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:33.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:33.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:33.768 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:33.768 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:33.768 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:33.769 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:33.769 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:33.769 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:34.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:34.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:34.125 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:34.125 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:34.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:34.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:34.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:34.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:34.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:34.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:34.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:34.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:34.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:35.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217767 2021-07-26T07:02:35.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217767 '!=' null 2021-07-26T07:02:35.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:35.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:35.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:35.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:35.275 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:35.276 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:35.276 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:35.276 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:35.278 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:35.278 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:35.278 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:35.279 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:35.279 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:35.279 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:35.279 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:35.279 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:35.280 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:35.280 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:35.280 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:35.281 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:35.281 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:35.281 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:35.281 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:35.282 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:35.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:35.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:35.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:35.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:35.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:35.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:35.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:35.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:35.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:35.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:35.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:35.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:35.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:36.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:36.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:36.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:36.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:36.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:36.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:36.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229447 2021-07-26T07:02:36.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229447 '!=' null 2021-07-26T07:02:36.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:36.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:36.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:36.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:36.951 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:36.951 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:36.952 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:36.952 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:36.952 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:36.952 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:36.953 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:36.953 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:36.953 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:36.953 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:36.954 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:36.954 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:36.954 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:36.954 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:36.955 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:36.955 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:36.955 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:36.955 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:36.956 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:36.956 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:36.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:37.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:37.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:37.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:37.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:37.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:37.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:37.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:37.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:37.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:37.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:37.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:37.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:37.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:37.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:37.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:37.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:37.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:37.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:37.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:37.965 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:37.965 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:38.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217583 2021-07-26T07:02:38.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217583 '!=' null 2021-07-26T07:02:38.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:38.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:38.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:38.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:38.522 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:38.522 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:38.523 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:38.523 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:38.523 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:38.523 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:38.524 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:38.524 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:38.524 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:38.524 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:38.524 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:38.525 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:38.525 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:38.525 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:38.525 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:38.525 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:38.526 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:38.526 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:38.526 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:38.526 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:38.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:38.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:38.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:38.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:38.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:38.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:38.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:38.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:38.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:38.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:38.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:38.997 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:38.997 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:39.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:39.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:39.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:39.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:39.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:39.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:39.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:39.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:39.733 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:40.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103324 2021-07-26T07:02:40.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103324 '!=' null 2021-07-26T07:02:40.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:40.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:40.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:40.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:40.382 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:40.383 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:40.383 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:40.383 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:40.384 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:40.384 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:40.384 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:40.384 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:40.385 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:40.385 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:40.385 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:40.385 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:40.385 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:40.386 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:40.386 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:40.386 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:40.386 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:40.387 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:40.387 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:40.387 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:40.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:40.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:40.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:40.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:40.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:40.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:40.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:40.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:40.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:40.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:40.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:40.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:40.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:41.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:41.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:41.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:41.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:41.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:41.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:41.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:41.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:41.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:41.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103323 2021-07-26T07:02:41.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103323 '!=' null 2021-07-26T07:02:41.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:41.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:41.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:41.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:42.045 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:42.045 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:42.046 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:42.046 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:42.046 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:42.046 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:42.047 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:42.047 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:42.047 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:42.047 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:42.047 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:42.048 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:42.048 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:42.048 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:42.048 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:42.049 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:42.049 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:42.049 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:42.049 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:42.049 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:42.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:42.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:42.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:42.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:42.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:42.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:42.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:42.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:42.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:42.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:42.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:42.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:42.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:42.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:42.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:42.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:42.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:42.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:42.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:42.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:42.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:42.948 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:43.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218114 2021-07-26T07:02:43.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218114 '!=' null 2021-07-26T07:02:43.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:43.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:43.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:43.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:43.547 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:43.548 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:43.548 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:43.548 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:43.548 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:43.548 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:43.549 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:43.549 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:43.549 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:43.549 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:43.550 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:43.550 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:43.550 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:43.550 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:43.550 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:43.551 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:43.551 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:43.551 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:43.551 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:43.552 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:43.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:43.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:43.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:43.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:43.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:43.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:43.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:43.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:43.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:44.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:44.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:44.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:44.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:44.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:44.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:44.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:44.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:44.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:44.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:44.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:44.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:44.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:45.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218114 2021-07-26T07:02:45.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218114 '!=' null 2021-07-26T07:02:45.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:45.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:45.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:45.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:45.309 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:45.309 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:45.309 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:45.309 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:45.310 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:45.310 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:45.310 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:45.310 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:45.310 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:45.311 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:45.311 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:45.311 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:45.311 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:45.312 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:45.312 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:45.312 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:45.312 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:45.312 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:45.313 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:45.313 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:45.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:45.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:45.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:45.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:45.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:45.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:45.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:45.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:45.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:46.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:46.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:46.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:46.041 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:46.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:46.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:46.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:46.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:46.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:46.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:46.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:46.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:46.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:47.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103363 2021-07-26T07:02:47.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103363 '!=' null 2021-07-26T07:02:47.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:47.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:47.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:47.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:47.190 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:47.191 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:47.192 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:47.192 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:47.192 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:47.192 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:47.192 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:47.193 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:47.193 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:47.193 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:47.193 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:47.194 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:47.194 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:47.194 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:47.194 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:47.195 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:47.195 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:47.195 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:47.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:47.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:47.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:47.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:47.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:47.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:47.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:47.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:47.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:47.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:47.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:47.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:47.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:48.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:48.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:48.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:48.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:48.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:48.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:48.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:48.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:48.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:48.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218330 2021-07-26T07:02:48.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218330 '!=' null 2021-07-26T07:02:48.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:48.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:48.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:48.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:48.706 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:48.706 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:48.707 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:48.708 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:48.709 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:48.710 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:48.710 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:48.710 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:48.710 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:48.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:48.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:48.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:48.814 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:48.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:48.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:48.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:48.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:48.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:49.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:49.175 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:49.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:49.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:49.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:49.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:49.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:49.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:49.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:49.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:49.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:49.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:49.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:50.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229770 2021-07-26T07:02:50.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229770 '!=' null 2021-07-26T07:02:50.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:50.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:50.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:50.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:50.324 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:50.324 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:50.324 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:50.324 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:50.325 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:50.325 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:50.325 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:50.325 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:50.326 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:50.326 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:50.326 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:50.326 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:50.326 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:50.327 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:50.327 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:50.327 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:50.327 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:50.328 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:50.328 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:50.328 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:50.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:50.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:50.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:50.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:50.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:50.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:50.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:50.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:50.435 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:50.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:50.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:50.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:50.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:51.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:51.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:51.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:51.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:51.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:51.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:51.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:51.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:51.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:51.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229905 2021-07-26T07:02:51.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229905 '!=' null 2021-07-26T07:02:51.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:51.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:51.770 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:51.770 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:51.852 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:51.852 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:51.853 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:51.853 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:51.853 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:51.853 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:51.853 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:51.854 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:51.854 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:51.854 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:51.854 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:51.855 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:51.855 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:51.855 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:51.855 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:51.855 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:51.856 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:51.857 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:51.857 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:51.857 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:51.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:51.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:51.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:51.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:51.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:51.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:51.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:51.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:51.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:52.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:52.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:52.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:52.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:52.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:52.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:52.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:52.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:52.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:52.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:52.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:52.740 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:52.740 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:53.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229905 2021-07-26T07:02:53.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229905 '!=' null 2021-07-26T07:02:53.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:53.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:53.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:53.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:53.327 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:53.328 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:53.328 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:53.328 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:53.328 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:53.329 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:53.329 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:53.329 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:53.329 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:53.329 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:53.330 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:53.330 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:53.330 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:53.330 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:53.331 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:53.331 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:53.331 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:53.331 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:53.331 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:53.332 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:53.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:53.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:53.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:53.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:53.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:53.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:53.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:53.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:53.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:53.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:53.830 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:53.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:53.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:54.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:54.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:54.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:54.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:54.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:54.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:54.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:54.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:54.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:55.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218450 2021-07-26T07:02:55.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218450 '!=' null 2021-07-26T07:02:55.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:55.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:55.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:55.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:55.220 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:55.220 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:55.221 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:55.221 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:55.221 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:55.222 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:55.222 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:55.222 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:55.222 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:55.222 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:55.223 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:55.223 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:55.223 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:55.223 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:55.224 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:55.224 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:55.224 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:55.224 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:55.224 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:55.225 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:55.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:55.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:55.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:55.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:55.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:55.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:55.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:55.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:55.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:55.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:55.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:55.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:55.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:56.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:56.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:56.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:56.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:56.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:56.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:56.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:56.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:56.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:56.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103515 2021-07-26T07:02:56.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103515 '!=' null 2021-07-26T07:02:56.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:56.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:56.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:56.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:56.904 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:56.905 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:56.905 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:56.905 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:56.906 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:56.906 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:56.906 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:56.906 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:56.906 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:56.907 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:56.907 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:56.907 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:56.907 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:56.907 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:56.908 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:56.908 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:56.908 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:56.908 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:56.909 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:56.909 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:56.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:57.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:57.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:57.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:57.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:57.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:57.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:57.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:57.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:57.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:57.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:57.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:57.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:58.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:58.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:58.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:58.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:58.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:58.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:58.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:58.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:58.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:02:58.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=20092460 2021-07-26T07:02:58.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 20092460 '!=' null 2021-07-26T07:02:58.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:02:58.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:58.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:02:58.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:02:58.796 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:02:58.797 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:02:58.797 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:58.797 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:58.797 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:58.798 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:02:58.798 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:02:58.798 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:02:58.798 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:02:58.799 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:02:58.799 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:58.799 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:02:58.799 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:02:58.799 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:02:58.800 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:02:58.800 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:02:58.800 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:02:58.800 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:02:58.801 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:02:58.801 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:02:58.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:02:58.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:02:58.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:02:58.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:02:58.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:02:58.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:02:58.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:02:58.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:02:58.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:02:59.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:02:59.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:02:59.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:02:59.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:02:59.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:02:59.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:02:59.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:02:59.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:02:59.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:02:59.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:02:59.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:02:59.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:02:59.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:00.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13977322 2021-07-26T07:03:00.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13977322 '!=' null 2021-07-26T07:03:00.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:00.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:00.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:00.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:00.433 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:00.434 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:00.434 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:00.434 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:00.435 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:00.435 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:00.435 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:00.435 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:00.435 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:00.436 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:00.436 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:00.436 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:00.436 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:00.436 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:00.437 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:00.437 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:00.437 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:00.437 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:00.438 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:00.438 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:00.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:00.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:00.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:00.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:00.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:00.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:00.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:00.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:00.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:01.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:01.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:01.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:01.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:01.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:01.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:01.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:01.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:01.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:01.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:01.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:01.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:01.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:02.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13977520 2021-07-26T07:03:02.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13977520 '!=' null 2021-07-26T07:03:02.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:02.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:02.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:02.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:02.392 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:02.393 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:02.393 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:02.393 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:02.394 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:02.394 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:02.394 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:02.394 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:02.394 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:02.395 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:02.396 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:02.396 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:02.396 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:02.396 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:02.397 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:02.397 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:02.397 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:02.397 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:02.398 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:02.398 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:02.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:02.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:02.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:02.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:02.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:02.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:02.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:02.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:02.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:02.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:02.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:02.866 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:02.866 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:03.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:03.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:03.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:03.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:03.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:03.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:03.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:03.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:03.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:04.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345411 2021-07-26T07:03:04.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345411 '!=' null 2021-07-26T07:03:04.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:04.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:04.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:04.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:04.442 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:04.442 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:04.443 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:04.443 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:04.443 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:04.443 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:04.443 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:04.444 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:04.444 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:04.444 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:04.444 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:04.444 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:04.445 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:04.445 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:04.445 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:04.445 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:04.446 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:04.446 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:04.446 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:04.446 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:04.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:04.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:04.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:04.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:04.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:04.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:04.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:04.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:04.554 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:04.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:04.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:04.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:04.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:05.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:05.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:05.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:05.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:05.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:05.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:05.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:05.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:05.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:05.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356655 2021-07-26T07:03:05.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356655 '!=' null 2021-07-26T07:03:05.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:05.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:05.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:05.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:06.037 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:06.037 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:06.038 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:06.038 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:06.038 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:06.038 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:06.039 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:06.039 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:06.039 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:06.039 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:06.040 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:06.040 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:06.040 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:06.040 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:06.040 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:06.041 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:06.041 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:06.041 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:06.041 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:06.041 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:06.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:06.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:06.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:06.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:06.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:06.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:06.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:06.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:06.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:06.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:06.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:06.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:06.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:07.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:07.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:07.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:07.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345297 2021-07-26T07:03:07.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345297 '!=' null 2021-07-26T07:03:07.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:07.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:07.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:07.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:07.707 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:07.707 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:07.708 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:07.708 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:07.708 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:07.708 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:07.709 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:07.709 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:07.709 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:07.709 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:07.709 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:07.710 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:07.710 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:07.710 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:07.710 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:07.711 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:07.711 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:07.711 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:07.711 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:07.711 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:07.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:07.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:07.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:07.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:07.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:07.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:07.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:07.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:07.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:08.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:08.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:08.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:08.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:08.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:08.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:08.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:08.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:08.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:08.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:08.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:08.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:08.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:09.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18345297 2021-07-26T07:03:09.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18345297 '!=' null 2021-07-26T07:03:09.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:09.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:09.330 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:09.330 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:09.413 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:09.413 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:09.414 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:09.415 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:09.415 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:09.415 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:09.415 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:09.416 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:09.416 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:09.416 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:09.416 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:09.416 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:09.417 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:09.417 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:09.417 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:09.417 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:09.418 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:09.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:09.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:09.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:09.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:09.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:09.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:09.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:09.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:09.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:09.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:09.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:09.874 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:09.875 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:10.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:10.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:10.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:10.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:10.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:10.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:10.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:10.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:10.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:10.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356282 2021-07-26T07:03:10.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356282 '!=' null 2021-07-26T07:03:10.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:10.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:10.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:10.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:11.073 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:11.074 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:11.074 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:11.074 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:11.074 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:11.075 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:11.075 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:11.075 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:11.075 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:11.076 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:11.076 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:11.076 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:11.076 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:11.076 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:11.077 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:11.077 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:11.077 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:11.077 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:11.077 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:11.078 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:11.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:11.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:11.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:11.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:11.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:11.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:11.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:11.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:11.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:11.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:11.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:11.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:11.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:11.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:11.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:11.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:11.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:11.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:11.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:11.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:11.983 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:11.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:12.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356238 2021-07-26T07:03:12.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356238 '!=' null 2021-07-26T07:03:12.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:12.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:12.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:12.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:12.700 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:12.701 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:12.701 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:12.701 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:12.701 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:12.702 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:12.702 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:12.702 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:12.702 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:12.702 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:12.703 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:12.703 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:12.703 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:12.703 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:12.704 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:12.704 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:12.704 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:12.704 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:12.704 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:12.705 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:12.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:12.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:12.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:12.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:12.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:12.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:12.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:12.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:12.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:13.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:13.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:13.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:13.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:13.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:13.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:13.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:13.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:13.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:13.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:13.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:13.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:13.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:14.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470408 2021-07-26T07:03:14.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470408 '!=' null 2021-07-26T07:03:14.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:14.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:14.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:14.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:14.518 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:14.518 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:14.519 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:14.519 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:14.519 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:14.519 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:14.520 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:14.520 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:14.520 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:14.520 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:14.521 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:14.521 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:14.521 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:14.521 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:14.521 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:14.522 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:14.522 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:14.522 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:14.522 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:14.523 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:14.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:14.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:14.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:14.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:14.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:14.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:14.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:14.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:14.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:14.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:14.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:14.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:14.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:15.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:15.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:15.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:15.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:15.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:15.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:15.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:15.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:15.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:16.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102628 2021-07-26T07:03:16.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102628 '!=' null 2021-07-26T07:03:16.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:16.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:16.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:16.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:16.228 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:16.228 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:16.228 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:16.229 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:16.229 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:16.229 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:16.229 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:16.230 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:16.230 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:16.230 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:16.230 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:16.230 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:16.231 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:16.231 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:16.231 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:16.231 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:16.232 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:16.232 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:16.232 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:16.232 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:16.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:16.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:16.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:16.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:16.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:16.339 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:16.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:16.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:16.340 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:16.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:16.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:16.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:17.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:17.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:17.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:17.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:17.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:17.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:17.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:17.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470127 2021-07-26T07:03:17.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470127 '!=' null 2021-07-26T07:03:17.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:17.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:17.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:17.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:17.945 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:17.945 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:17.946 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:17.946 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:17.946 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:17.946 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:17.947 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:17.947 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:17.947 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:17.947 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:17.948 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:17.948 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:17.948 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:17.948 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:17.948 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:17.949 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:17.949 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:17.949 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:17.949 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:17.950 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:17.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:18.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:18.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:18.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:18.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:18.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:18.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:18.057 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:18.057 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:18.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:18.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:18.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:18.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:18.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:18.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:18.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:18.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:18.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:18.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:18.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:18.855 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:18.855 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:19.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482014 2021-07-26T07:03:19.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482014 '!=' null 2021-07-26T07:03:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:19.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:19.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:19.820 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:19.821 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:19.821 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:19.821 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:19.821 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:19.821 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:19.822 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:19.822 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:19.822 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:19.822 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:19.823 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:19.823 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:19.823 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:19.823 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:19.823 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:19.824 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:19.824 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:19.824 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:19.825 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:19.825 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:19.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:19.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:19.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:19.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:19.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:19.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:19.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:19.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:19.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:20.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:20.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:20.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:20.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:20.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:20.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:20.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:20.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:20.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:20.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:20.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:20.805 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:20.805 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:21.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482014 2021-07-26T07:03:21.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482014 '!=' null 2021-07-26T07:03:21.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:21.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:21.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:21.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:21.382 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:21.383 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:21.383 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:21.383 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:21.383 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:21.384 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:21.384 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:21.384 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:21.384 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:21.384 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:21.385 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:21.385 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:21.385 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:21.385 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:21.386 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:21.386 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:21.386 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:21.386 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:21.387 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:21.387 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:21.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:21.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:21.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:21.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:21.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:21.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:21.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:21.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:21.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:21.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:21.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:21.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:21.892 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:22.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:22.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:22.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:22.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:22.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:22.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:22.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:22.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:22.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:22.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=10482123 2021-07-26T07:03:22.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 10482123 '!=' null 2021-07-26T07:03:22.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:22.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:22.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:22.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:22.906 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:22.906 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:22.906 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:22.906 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:22.907 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:22.907 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:22.907 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:22.907 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:22.908 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:22.908 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:22.908 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:22.908 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:22.908 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:22.909 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:22.909 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:22.909 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:22.909 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:22.910 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:22.910 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:22.910 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:22.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:23.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:23.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:23.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:23.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:23.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:23.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:23.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:23.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:23.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:23.368 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:23.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:23.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:23.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:23.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:23.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:23.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:23.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:23.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:23.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:23.890 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:23.890 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:24.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470470 2021-07-26T07:03:24.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470470 '!=' null 2021-07-26T07:03:24.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:24.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:24.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:24.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:24.492 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:24.492 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:24.495 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:24.495 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:24.495 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:24.495 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:24.496 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:24.496 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:24.496 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:24.496 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:24.496 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:24.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:24.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:24.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:24.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:24.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:24.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:24.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:24.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:24.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:24.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:24.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:24.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:24.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:25.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:25.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:25.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:25.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:25.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:25.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:25.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:25.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:25.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:25.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356036 2021-07-26T07:03:25.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356036 '!=' null 2021-07-26T07:03:25.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:25.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:25.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:25.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:25.903 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:25.904 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:25.904 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:25.904 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:25.904 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:25.905 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:25.905 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:25.905 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:25.905 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:25.905 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:25.906 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:25.906 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:25.906 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:25.906 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:25.907 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:25.907 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:25.907 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:25.907 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:25.907 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:25.908 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:25.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:26.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:26.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:26.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:26.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:26.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:26.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:26.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:26.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:26.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:26.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:26.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:26.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:26.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:26.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:26.883 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:27.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356036 2021-07-26T07:03:27.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356036 '!=' null 2021-07-26T07:03:27.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:27.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:27.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:27.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:27.430 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:27.430 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:27.431 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:27.431 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:27.431 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:27.431 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:27.432 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:27.432 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:27.432 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:27.432 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:27.433 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:27.433 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:27.433 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:27.433 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:27.433 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:27.434 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:27.434 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:27.434 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:27.434 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:27.434 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:27.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:27.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:27.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:27.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:27.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:27.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:27.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:27.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:27.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:27.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:27.983 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:27.983 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:27.983 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:28.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:28.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:28.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:28.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:28.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:28.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:28.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:28.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:28.668 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:29.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344379 2021-07-26T07:03:29.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344379 '!=' null 2021-07-26T07:03:29.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:29.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:29.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:29.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:29.309 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:29.309 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:29.309 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:29.310 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:29.310 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:29.310 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:29.310 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:29.310 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:29.311 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:29.311 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:29.311 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:29.311 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:29.312 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:29.312 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:29.312 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:29.312 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:29.312 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:29.313 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:29.313 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:29.313 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:29.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:29.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:29.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:29.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:29.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:29.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:29.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:29.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:29.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:29.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:29.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:29.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:29.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:30.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:30.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:30.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:30.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:30.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:30.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:30.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:30.263 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:30.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:30.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976704 2021-07-26T07:03:30.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976704 '!=' null 2021-07-26T07:03:30.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:30.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:30.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:30.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:30.894 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:30.895 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:30.895 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:30.895 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:30.895 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:30.895 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:30.896 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:30.896 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:30.896 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:30.896 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:30.897 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:30.897 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:30.897 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:30.897 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:30.897 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:30.898 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:30.898 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:30.898 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:30.898 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:30.898 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:30.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:31.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:31.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:31.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:31.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:31.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:31.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:31.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:31.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:31.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:31.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:31.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:31.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:32.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:32.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:32.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:32.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:32.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:32.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:32.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:32.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:32.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:32.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13976869 2021-07-26T07:03:32.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13976869 '!=' null 2021-07-26T07:03:32.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:32.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:32.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:32.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:32.752 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:32.753 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:32.753 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:32.753 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:32.753 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:32.754 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:32.754 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:32.754 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:32.755 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:32.755 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:32.756 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:32.756 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:32.756 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:32.756 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:32.757 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:32.757 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:32.757 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:32.757 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:32.757 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:32.758 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:32.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:32.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:32.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:32.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:32.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:32.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:32.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:32.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:32.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:33.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:33.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:33.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:33.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:33.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:33.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:33.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:33.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:33.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:33.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:33.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:33.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:33.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:34.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218500 2021-07-26T07:03:34.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218500 '!=' null 2021-07-26T07:03:34.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:34.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:34.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:34.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:34.401 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:34.402 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:34.402 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:34.402 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:34.402 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:34.403 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:34.403 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:34.403 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:34.403 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:34.404 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:34.404 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:34.405 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:34.405 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:34.405 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:34.405 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:34.406 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:34.406 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:34.406 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:34.406 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:34.407 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:34.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:34.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:34.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:34.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:34.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:34.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:34.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:34.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:34.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:34.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:34.911 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:34.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:34.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:35.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:35.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:35.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:35.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:35.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:35.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:35.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:35.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:35.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:36.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229800 2021-07-26T07:03:36.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229800 '!=' null 2021-07-26T07:03:36.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:36.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:36.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:36.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:36.101 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:36.102 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:36.102 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:36.102 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:36.102 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:36.102 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:36.103 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:36.103 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:36.103 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:36.103 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:36.104 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:36.104 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:36.104 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:36.104 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:36.105 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:36.105 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:36.105 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:36.105 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:36.105 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:36.106 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:36.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:36.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:36.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:36.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:36.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:36.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:36.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:36.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:36.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:36.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:36.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:36.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:36.565 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:37.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:37.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:37.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:37.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:37.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:37.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:37.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:37.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:37.013 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:37.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218355 2021-07-26T07:03:37.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218355 '!=' null 2021-07-26T07:03:37.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:37.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:37.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:37.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:38.034 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:38.035 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:38.035 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:38.035 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:38.036 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:38.036 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:38.036 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:38.036 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:38.037 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:38.037 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:38.037 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:38.037 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:38.038 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:38.038 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:38.038 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:38.038 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:38.039 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:38.039 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:38.039 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:38.040 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:38.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:38.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:38.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:38.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:38.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:38.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:38.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:38.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:38.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:38.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:38.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:38.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:38.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:38.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:38.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:38.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:38.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:39.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218355 2021-07-26T07:03:39.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218355 '!=' null 2021-07-26T07:03:39.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:39.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:39.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:39.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:39.637 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:39.638 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:39.638 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:39.638 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:39.638 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:39.638 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:39.639 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:39.639 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:39.639 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:39.639 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:39.640 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:39.640 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:39.640 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:39.640 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:39.640 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:39.641 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:39.641 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:39.641 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:39.642 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:39.642 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:39.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:39.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:39.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:39.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:39.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:39.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:39.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:39.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:39.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:40.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:40.106 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:40.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:40.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:40.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:40.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:40.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:40.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:40.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:40.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:40.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:40.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:40.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:41.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229837 2021-07-26T07:03:41.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229837 '!=' null 2021-07-26T07:03:41.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:41.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:41.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:41.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:41.474 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:41.475 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:41.475 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:41.475 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:41.475 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:41.476 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:41.476 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:41.476 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:41.476 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:41.477 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:41.477 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:41.477 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:41.477 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:41.477 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:41.478 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:41.478 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:41.478 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:41.478 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:41.478 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:41.479 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:41.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:41.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:41.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:41.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:41.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:41.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:41.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:41.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:41.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:41.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:41.957 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:41.957 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:41.957 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:42.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:42.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:42.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:42.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:42.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:42.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:42.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:42.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:42.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:43.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12230052 2021-07-26T07:03:43.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12230052 '!=' null 2021-07-26T07:03:43.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:43.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:43.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:43.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:43.098 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:43.098 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:43.098 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:43.098 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:43.099 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:43.099 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:43.099 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:43.099 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:43.100 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:43.100 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:43.100 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:43.100 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:43.100 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:43.101 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:43.101 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:43.101 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:43.101 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:43.102 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:43.102 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:43.102 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:43.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:43.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:43.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:43.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:43.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:43.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:43.208 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:43.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:43.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:43.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:43.750 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:43.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:43.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:44.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:44.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:44.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:44.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:44.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:44.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:44.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:44.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:44.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:44.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=18344967 2021-07-26T07:03:44.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 18344967 '!=' null 2021-07-26T07:03:44.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:44.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:44.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:44.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:44.881 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:44.881 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:44.881 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:44.882 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:44.882 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:44.882 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:44.882 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:44.883 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:44.883 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:44.883 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:44.883 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:44.883 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:44.884 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:44.884 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:44.884 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:44.884 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:44.885 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:44.885 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:44.885 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:44.885 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:44.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:44.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:44.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:44.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:44.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:44.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:44.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:44.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:44.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:45.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:45.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:45.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:45.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:45.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:45.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:45.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:45.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:45.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:45.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:45.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:45.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:45.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:46.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13103353 2021-07-26T07:03:46.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13103353 '!=' null 2021-07-26T07:03:46.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:46.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:46.353 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:46.353 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:46.437 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:46.438 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:46.438 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:46.438 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:46.438 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:46.439 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:46.439 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:46.439 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:46.439 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:46.439 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:46.440 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:46.440 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:46.440 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:46.440 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:46.440 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:46.441 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:46.441 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:46.441 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:46.441 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:46.442 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:46.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:46.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:46.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:46.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:46.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:46.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:46.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:46.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:46.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:46.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:46.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:46.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:46.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:47.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:47.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:47.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:47.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:47.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:47.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:47.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:47.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:47.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:47.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218256 2021-07-26T07:03:47.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218256 '!=' null 2021-07-26T07:03:47.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:47.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:47.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:47.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:48.041 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:48.042 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:48.042 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:48.042 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:48.042 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:48.043 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:48.043 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:48.043 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:48.043 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:48.043 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:48.044 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:48.045 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:48.045 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:48.045 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:48.045 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:48.046 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:48.046 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:48.046 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:48.046 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:48.046 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:48.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:48.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:48.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:48.151 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:48.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:48.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:48.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:48.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:48.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:48.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:48.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:48.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:48.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:49.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:49.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:49.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:49.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:49.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:49.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:49.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:49.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:49.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:50.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19218256 2021-07-26T07:03:50.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19218256 '!=' null 2021-07-26T07:03:50.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:50.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:50.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:50.144 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:50.222 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:50.222 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:50.222 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:50.222 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:50.223 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:50.223 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:50.223 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:50.223 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:50.224 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:50.224 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:50.224 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:50.224 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:50.224 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:50.225 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:50.225 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:50.225 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:50.225 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:50.225 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:50.226 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:50.226 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:50.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:50.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:50.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:50.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:50.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:50.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:50.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:50.334 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:50.334 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:50.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:50.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:50.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:50.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:51.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:51.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:51.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:51.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:51.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:51.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:51.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:51.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:51.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:51.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229536 2021-07-26T07:03:51.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229536 '!=' null 2021-07-26T07:03:51.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:51.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:51.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:51.794 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:51.879 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:51.880 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:51.880 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:51.880 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:51.880 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:51.881 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:51.881 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:51.881 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:51.881 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:51.882 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:51.882 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:51.882 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:51.882 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:51.882 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:51.883 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:51.883 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:51.883 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:51.883 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:51.884 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:51.884 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:51.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:51.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:51.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:51.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:51.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:51.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:51.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:51.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:51.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:52.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:52.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:52.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:52.359 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:52.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:52.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:52.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:52.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:52.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:52.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:52.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:52.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:52.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:53.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229536 2021-07-26T07:03:53.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229536 '!=' null 2021-07-26T07:03:53.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:53.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:53.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:53.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:53.504 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:53.505 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:53.505 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:53.505 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:53.505 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:53.506 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:53.506 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:53.506 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:53.506 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:53.506 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:53.507 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:53.507 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:53.507 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:53.507 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:53.507 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:53.508 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:53.508 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:53.508 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:53.508 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:53.508 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:53.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:53.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:53.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:53.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:53.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:53.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:53.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:53.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:53.618 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:54.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:54.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:54.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:54.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:54.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:54.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:54.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:54.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:54.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:54.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:54.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:54.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:54.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:54.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217508 2021-07-26T07:03:54.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217508 '!=' null 2021-07-26T07:03:54.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:54.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:54.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:54.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:55.082 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:55.083 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:55.083 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:55.083 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:55.084 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:55.084 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:55.084 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:55.084 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:55.084 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:55.085 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:55.085 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:55.085 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:55.086 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:55.086 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:55.086 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:55.086 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:55.087 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:55.087 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:55.087 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:55.087 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:55.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:55.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:55.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:55.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:55.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:55.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:55.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:55.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:55.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:55.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:55.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:55.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:55.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:56.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:56.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:56.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:56.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:56.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:56.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:56.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:56.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:56.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:56.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102685 2021-07-26T07:03:56.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102685 '!=' null 2021-07-26T07:03:56.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:56.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:56.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:56.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:56.874 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:56.874 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:56.874 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:56.875 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:56.875 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:56.875 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:56.875 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:56.876 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:56.876 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:56.876 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:56.876 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:56.876 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:56.877 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:56.877 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:56.877 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:56.877 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:56.878 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:56.878 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:56.878 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:56.878 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:56.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:56.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:56.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:56.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:56.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:56.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:56.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:56.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:56.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:57.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:57.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:57.316 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:57.316 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:57.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:57.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:57.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:57.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:57.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:57.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:57.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:57.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:57.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:03:58.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217121 2021-07-26T07:03:58.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217121 '!=' null 2021-07-26T07:03:58.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:03:58.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:58.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:03:58.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:03:58.526 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:03:58.527 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:03:58.527 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:58.528 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:58.528 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:58.528 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:03:58.529 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:03:58.529 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:03:58.530 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:03:58.530 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:03:58.530 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:58.531 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:03:58.531 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:03:58.531 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:03:58.531 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:03:58.532 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:03:58.532 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:03:58.532 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:03:58.532 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:03:58.533 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:03:58.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:03:58.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:03:58.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:03:58.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:03:58.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:03:58.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:03:58.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:03:58.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:03:58.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:03:59.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:03:59.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:03:59.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:03:59.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:03:59.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:03:59.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:03:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:03:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:03:59.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:03:59.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:03:59.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:03:59.489 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:03:59.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:00.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102585 2021-07-26T07:04:00.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102585 '!=' null 2021-07-26T07:04:00.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:00.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:00.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:00.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:00.095 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:00.095 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:00.095 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:00.096 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:00.096 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:00.096 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:00.096 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:00.097 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:00.097 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:00.097 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:00.097 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:00.097 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:00.098 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:00.098 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:00.098 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:00.098 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:00.099 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:00.099 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:00.099 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:00.099 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:00.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:00.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:00.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:00.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:00.205 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:00.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:00.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:00.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:00.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:00.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:00.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:00.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:00.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:01.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:01.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:01.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:01.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:01.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:01.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:01.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:01.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:01.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:01.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102946 2021-07-26T07:04:01.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102946 '!=' null 2021-07-26T07:04:01.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:01.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:01.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:01.539 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:01.618 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:01.619 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:01.619 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:01.619 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:01.619 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:01.620 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:01.620 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:01.620 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:01.620 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:01.620 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:01.621 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:01.621 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:01.621 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:01.621 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:01.621 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:01.622 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:01.622 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:01.622 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:01.622 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:01.623 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:01.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:01.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:01.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:01.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:01.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:01.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:01.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:01.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:01.734 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:02.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:02.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:02.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:02.171 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:02.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:02.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:02.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:02.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:02.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:02.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:02.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:02.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:02.665 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:03.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=13102946 2021-07-26T07:04:03.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 13102946 '!=' null 2021-07-26T07:04:03.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:03.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:03.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:03.108 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:03.187 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:03.187 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:03.187 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:03.187 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:03.188 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:03.188 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:03.188 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:03.188 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:03.189 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:03.189 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:03.189 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:03.189 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:03.189 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:03.190 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:03.190 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:03.190 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:03.190 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:03.191 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:03.191 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:03.191 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:03.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:03.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:03.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:03.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:03.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:03.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:03.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:03.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:03.299 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:03.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:03.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:03.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:03.662 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:04.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:04.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:04.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:04.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:04.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:04.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:04.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:04.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:04.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:04.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=19217701 2021-07-26T07:04:04.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 19217701 '!=' null 2021-07-26T07:04:04.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:04.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:04.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:04.627 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:04.696 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:04.696 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:04.697 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:04.697 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:04.697 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:04.697 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:04.698 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:04.698 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:04.698 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:04.698 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:04.699 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:04.699 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:04.699 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:04.699 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:04.699 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:04.700 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:04.700 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:04.700 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:04.700 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:04.700 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:04.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:04.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:04.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:04.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:04.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:04.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:04.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:04.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:04.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:05.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:05.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:05.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:05.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:05.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:05.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:05.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:05.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:05.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:05.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:05.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:05.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:05.784 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:06.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=12229425 2021-07-26T07:04:06.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 12229425 '!=' null 2021-07-26T07:04:06.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:06.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:06.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:06.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:06.464 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:06.465 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:06.465 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:06.465 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:06.465 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:06.466 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:06.466 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:06.466 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:06.466 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:06.467 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:06.467 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:06.467 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:06.467 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:06.467 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:06.468 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:06.468 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:06.468 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:06.468 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:06.468 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:06.469 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:06.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:06.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:06.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:06.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:06.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:06.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:06.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:06.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:06.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:07.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:07.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:07.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:07.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:07.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:07.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:07.647 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:07.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:08.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470836 2021-07-26T07:04:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470836 '!=' null 2021-07-26T07:04:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:08.436 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:08.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:08.519 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:08.520 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:08.520 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:08.520 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:08.521 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:08.521 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:08.521 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:08.522 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:08.522 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:08.522 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:08.522 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:08.523 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:08.523 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:08.523 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:08.523 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:08.523 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:08.524 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:08.524 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:08.524 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:08.524 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:08.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:08.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:08.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:08.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:08.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:08.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:08.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:08.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:08.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:08.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:08.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:08.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:08.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:09.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:09.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:09.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:09.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:09.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:09.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:09.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:09.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:09.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:09.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356152 2021-07-26T07:04:09.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356152 '!=' null 2021-07-26T07:04:09.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:09.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:09.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:09.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:10.061 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:10.061 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:10.061 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:10.061 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:10.062 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:10.062 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:10.062 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:10.062 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:10.063 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:10.063 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:10.063 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:10.063 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:10.064 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:10.064 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:10.064 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:10.064 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:10.064 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:10.065 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:10.065 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:10.065 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:10.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:10.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:10.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:10.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:10.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:10.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:10.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:10.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:10.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:10.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:10.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:10.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:10.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:11.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:11.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:11.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:11.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:11.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:11.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:11.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:11.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:11.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:11.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356105 2021-07-26T07:04:11.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356105 '!=' null 2021-07-26T07:04:11.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:11.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:11.669 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:11.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:11.759 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:11.759 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:11.759 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:11.760 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:11.760 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:11.760 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:11.760 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:11.761 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:11.761 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:11.761 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:11.761 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:11.761 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:11.762 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:11.762 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:11.762 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:11.762 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:11.763 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:11.763 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:11.763 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:11.763 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:11.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:11.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:11.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:11.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:11.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:11.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:11.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:11.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:11.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:12.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:12.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:12.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:12.240 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:12.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:12.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:12.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:12.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:12.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:12.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:12.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:12.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:12.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:13.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11356105 2021-07-26T07:04:13.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11356105 '!=' null 2021-07-26T07:04:13.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:13.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:13.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:13.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:13.371 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:13.371 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:13.372 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:13.372 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:13.372 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:13.372 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:13.372 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:13.373 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:13.373 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:13.373 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:13.373 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:13.373 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:13.374 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:13.374 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:13.375 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:13.375 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:13.375 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:13.375 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:13.376 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:13.376 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:13.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:13.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:13.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:13.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:13.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:13.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:13.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:13.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:13.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:13.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:13.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:13.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:13.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:14.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:14.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:14.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:14.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:14.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:14.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:14.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:14.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:14.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:15.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470611 2021-07-26T07:04:15.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470611 '!=' null 2021-07-26T07:04:15.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:15.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:15.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:15.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:15.100 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:15.100 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:15.101 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:15.101 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:15.102 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:15.102 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:15.103 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:15.103 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:15.104 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:15.104 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:15.105 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:15.105 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:15.106 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:15.106 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:15.107 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:15.107 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:15.108 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:15.108 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:15.109 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:15.109 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:15.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:15.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:15.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:15.210 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:15.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:15.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:15.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:15.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:15.211 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:15.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:15.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:15.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:15.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:16.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:16.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:16.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:16.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:16.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:16.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:16.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:16.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:16.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:16.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=11355886 2021-07-26T07:04:16.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 11355886 '!=' null 2021-07-26T07:04:16.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:16.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:16.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:16.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:16.680 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:16.681 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:16.681 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:16.681 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:16.682 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:16.682 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:16.682 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:16.682 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:16.682 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:16.683 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:16.689 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:16.690 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:16.690 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:16.690 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:16.690 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:16.690 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:16.691 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:16.691 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:16.691 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:16.692 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:16.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:16.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:16.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:16.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:16.797 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:16.797 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:16.797 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:16.797 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:16.798 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:17.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:17.501 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:17.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:17.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:17.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:17.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:17.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:17.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:17.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:17.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:17.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:17.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:17.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:18.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=17470562 2021-07-26T07:04:18.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 17470562 '!=' null 2021-07-26T07:04:18.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:18.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:18.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:18.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:18.476 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:18.476 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:18.476 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:18.476 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:18.477 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:18.477 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [ 2021-07-26T07:04:18.477 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:18.477 INFO:tasks.workunit.client.0.smithi094.stdout: "item": "3.0", 2021-07-26T07:04:18.477 INFO:tasks.workunit.client.0.smithi094.stdout: "prio": 190, 2021-07-26T07:04:18.478 INFO:tasks.workunit.client.0.smithi094.stdout: "can_preempt": true 2021-07-26T07:04:18.478 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:18.478 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:18.478 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:18.479 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:18.479 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:18.479 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:18.479 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:18.479 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:18.480 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:18.480 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:18.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:18.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:18.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:18.586 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:18.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:18.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:18.587 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:18.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:18.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:18.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:18.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:18.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:18.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:19.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:19.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:19.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:19.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:19.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:19.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:19.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:19.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:19.494 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:19.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=14675185 2021-07-26T07:04:19.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 14675185 '!=' null 2021-07-26T07:04:19.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:19.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:19.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:19.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:20.057 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:20.057 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:20.057 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:20.058 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:20.058 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:20.058 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:20.058 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:20.059 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:20.059 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:20.059 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:20.060 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:20.060 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:20.060 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:20.060 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:20.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:20.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:20.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:20.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:20.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:20.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:20.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:20.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:20.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:20.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:20.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:20.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:20.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:21.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:21.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:21.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:21.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:21.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:21.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:21.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:21.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:21.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:21.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=9172225 2021-07-26T07:04:21.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 9172225 '!=' null 2021-07-26T07:04:21.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:21.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:21.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:21.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:21.677 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:21.677 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:21.678 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:21.678 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:21.678 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:21.678 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:21.678 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:21.679 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:21.679 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:21.679 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:21.679 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:21.680 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:21.680 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:21.680 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:21.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:21.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:21.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:21.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:21.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:21.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:21.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:21.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:21.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:22.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=2 2021-07-26T07:04:22.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:22.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:22.114 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 2 = 3 2021-07-26T07:04:22.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 2 '!=' 2 2021-07-26T07:04:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:04:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:04:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:04:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:04:22.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:04:22.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:04:22.532 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:04:23.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=9172225 2021-07-26T07:04:23.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 9172225 '!=' null 2021-07-26T07:04:23.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:04:23.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:23.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: CEPH_ARGS= 2021-07-26T07:04:23.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_recovery_reservations 2021-07-26T07:04:23.104 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:23.104 INFO:tasks.workunit.client.0.smithi094.stdout: "local_reservations": { 2021-07-26T07:04:23.104 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:23.105 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:23.105 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:23.105 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:23.105 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:23.105 INFO:tasks.workunit.client.0.smithi094.stdout: "remote_reservations": { 2021-07-26T07:04:23.106 INFO:tasks.workunit.client.0.smithi094.stdout: "max_allowed": 1, 2021-07-26T07:04:23.106 INFO:tasks.workunit.client.0.smithi094.stdout: "min_priority": 0, 2021-07-26T07:04:23.106 INFO:tasks.workunit.client.0.smithi094.stdout: "queues": [], 2021-07-26T07:04:23.106 INFO:tasks.workunit.client.0.smithi094.stdout: "in_progress": [] 2021-07-26T07:04:23.107 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:23.107 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:23.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:04:23.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:04:23.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:04:23.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:04:23.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:04:23.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:04:23.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:04:23.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:04:23.214 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:04:23.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=3 2021-07-26T07:04:23.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:04:23.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:04:23.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:04:23.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 3 = 3 2021-07-26T07:04:23.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:04:23.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:04:23.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:295: TEST_recovery_priority: ceph pg dump pgs 2021-07-26T07:04:24.320 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T07:04:24.321 INFO:tasks.workunit.client.0.smithi094.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T06:59:35.998453+0000 95'600 119:1090 [1,4] 1 [1,4] 1 0'0 2021-07-26T06:49:18.538563+0000 0'0 2021-07-26T06:49:18.538563+0000 0 2021-07-26T07:04:24.321 INFO:tasks.workunit.client.0.smithi094.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T07:04:19.464132+0000 95'600 120:1102 [1,2] 1 [1,2] 1 0'0 2021-07-26T06:49:06.707131+0000 0'0 2021-07-26T06:49:06.707131+0000 0 2021-07-26T07:04:24.322 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-07-26T07:01:58.792602+0000 95'600 119:1103 [1,2] 1 [1,2] 1 0'0 2021-07-26T06:49:01.157992+0000 0'0 2021-07-26T06:49:01.157992+0000 0 2021-07-26T07:04:24.322 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T07:04:24.322 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T07:04:24.323 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T07:04:24.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: get_asok_path osd.1 2021-07-26T07:04:24.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-07-26T07:04:24.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-07-26T07:04:24.333 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:04:24.334 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:24.334 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T07:04:24.334 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.175221/ceph-osd.1.asok 2021-07-26T07:04:24.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: CEPH_ARGS= 2021-07-26T07:04:24.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.175221/ceph-osd.1.asok dump_pgstate_history 2021-07-26T07:04:24.409 INFO:tasks.workunit.client.0.smithi094.stdout:{ 2021-07-26T07:04:24.410 INFO:tasks.workunit.client.0.smithi094.stdout: "pgs": [ 2021-07-26T07:04:24.410 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.410 INFO:tasks.workunit.client.0.smithi094.stdout: "pg": "7.0", 2021-07-26T07:04:24.410 INFO:tasks.workunit.client.0.smithi094.stdout: "currently": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.411 INFO:tasks.workunit.client.0.smithi094.stdout: "history": [ 2021-07-26T07:04:24.411 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.411 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "64", 2021-07-26T07:04:24.411 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.411 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.412 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.412 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:18.971289+0000", 2021-07-26T07:04:24.412 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:18.971310+0000" 2021-07-26T07:04:24.412 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.413 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.413 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.413 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:18.971324+0000", 2021-07-26T07:04:24.413 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:19.116924+0000" 2021-07-26T07:04:24.414 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.414 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.414 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.415 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:19.116926+0000", 2021-07-26T07:04:24.415 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:19.117099+0000" 2021-07-26T07:04:24.415 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.415 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.415 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.416 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:19.117100+0000", 2021-07-26T07:04:24.416 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:19.117124+0000" 2021-07-26T07:04:24.416 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.416 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.417 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.417 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:19.117124+0000", 2021-07-26T07:04:24.417 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:19.452675+0000" 2021-07-26T07:04:24.417 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.417 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.418 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.418 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:18.971317+0000", 2021-07-26T07:04:24.418 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:19.452690+0000" 2021-07-26T07:04:24.418 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.419 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.419 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.419 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:19.453164+0000", 2021-07-26T07:04:24.419 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.004730+0000" 2021-07-26T07:04:24.419 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.420 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.420 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.420 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.004731+0000", 2021-07-26T07:04:24.420 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.004890+0000" 2021-07-26T07:04:24.420 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.421 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.421 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.421 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.004891+0000", 2021-07-26T07:04:24.422 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.582340+0000" 2021-07-26T07:04:24.422 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.422 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.422 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.423 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:19.452691+0000", 2021-07-26T07:04:24.423 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.582381+0000" 2021-07-26T07:04:24.423 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.423 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.424 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.424 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:18.971311+0000", 2021-07-26T07:04:24.424 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.582417+0000" 2021-07-26T07:04:24.424 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.424 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.425 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.425 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:18.971282+0000", 2021-07-26T07:04:24.425 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.582424+0000" 2021-07-26T07:04:24.425 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.426 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.426 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.426 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.426 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "64", 2021-07-26T07:04:24.426 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.427 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.427 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.427 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.582425+0000", 2021-07-26T07:04:24.427 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.583193+0000" 2021-07-26T07:04:24.428 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.428 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.428 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.428 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.428 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "94", 2021-07-26T07:04:24.429 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.429 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.429 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.429 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.583206+0000", 2021-07-26T07:04:24.430 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.583243+0000" 2021-07-26T07:04:24.430 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.430 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.430 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.430 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.583267+0000", 2021-07-26T07:04:24.431 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.752623+0000" 2021-07-26T07:04:24.431 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.431 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.431 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.432 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.752625+0000", 2021-07-26T07:04:24.432 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.752838+0000" 2021-07-26T07:04:24.432 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.432 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.432 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.433 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.752839+0000", 2021-07-26T07:04:24.433 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:20.752863+0000" 2021-07-26T07:04:24.433 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.433 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.434 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.434 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.752864+0000", 2021-07-26T07:04:24.434 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:21.010302+0000" 2021-07-26T07:04:24.434 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.434 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.435 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.435 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.583256+0000", 2021-07-26T07:04:24.435 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:21.010317+0000" 2021-07-26T07:04:24.435 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.436 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.436 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.436 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:21.010515+0000", 2021-07-26T07:04:24.437 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:21.266988+0000" 2021-07-26T07:04:24.437 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.437 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.437 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.438 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:21.266989+0000", 2021-07-26T07:04:24.438 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:21.267092+0000" 2021-07-26T07:04:24.438 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.439 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.439 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.439 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:21.267093+0000", 2021-07-26T07:04:24.439 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.242351+0000" 2021-07-26T07:04:24.439 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.440 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.440 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.440 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:21.010318+0000", 2021-07-26T07:04:24.440 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.242408+0000" 2021-07-26T07:04:24.441 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.441 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.441 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.441 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.583244+0000", 2021-07-26T07:04:24.441 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.242451+0000" 2021-07-26T07:04:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:20.583194+0000", 2021-07-26T07:04:24.442 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.242458+0000" 2021-07-26T07:04:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.443 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "94", 2021-07-26T07:04:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.242460+0000", 2021-07-26T07:04:24.444 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.243055+0000" 2021-07-26T07:04:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.445 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.446 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "100", 2021-07-26T07:04:24.446 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.446 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.446 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.446 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243064+0000", 2021-07-26T07:04:24.447 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.243092+0000" 2021-07-26T07:04:24.447 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.447 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.447 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.447 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243114+0000", 2021-07-26T07:04:24.448 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.243390+0000" 2021-07-26T07:04:24.448 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.448 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.448 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.449 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243392+0000", 2021-07-26T07:04:24.449 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.243509+0000" 2021-07-26T07:04:24.449 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.449 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.449 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.450 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243510+0000", 2021-07-26T07:04:24.450 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.243534+0000" 2021-07-26T07:04:24.450 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.450 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.451 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.451 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243535+0000", 2021-07-26T07:04:24.451 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.544398+0000" 2021-07-26T07:04:24.451 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.451 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.452 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.452 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243106+0000", 2021-07-26T07:04:24.452 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.544413+0000" 2021-07-26T07:04:24.452 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.452 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.453 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.453 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.544578+0000", 2021-07-26T07:04:24.453 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.628846+0000" 2021-07-26T07:04:24.453 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.454 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.454 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.454 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.628848+0000", 2021-07-26T07:04:24.454 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:50.629026+0000" 2021-07-26T07:04:24.454 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.455 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.455 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.455 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.629027+0000", 2021-07-26T07:04:24.455 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.729172+0000" 2021-07-26T07:04:24.456 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.456 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.456 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.456 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.544414+0000", 2021-07-26T07:04:24.456 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.729257+0000" 2021-07-26T07:04:24.457 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.457 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.457 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.457 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243093+0000", 2021-07-26T07:04:24.457 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.729503+0000" 2021-07-26T07:04:24.458 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.458 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.458 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.458 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:50.243055+0000", 2021-07-26T07:04:24.459 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.729512+0000" 2021-07-26T07:04:24.459 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.459 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.459 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.459 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.460 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "101", 2021-07-26T07:04:24.460 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.460 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.460 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.461 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.729513+0000", 2021-07-26T07:04:24.461 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.730530+0000" 2021-07-26T07:04:24.461 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.461 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.461 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.462 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.462 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "102", 2021-07-26T07:04:24.462 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.462 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.462 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.463 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.730546+0000", 2021-07-26T07:04:24.463 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.730582+0000" 2021-07-26T07:04:24.463 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.463 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.464 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.464 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.730606+0000", 2021-07-26T07:04:24.464 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.821776+0000" 2021-07-26T07:04:24.464 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.464 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.465 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.465 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.821778+0000", 2021-07-26T07:04:24.465 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.821916+0000" 2021-07-26T07:04:24.465 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.465 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.466 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.466 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.730592+0000", 2021-07-26T07:04:24.466 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:19.821934+0000" 2021-07-26T07:04:24.466 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.466 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.467 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/WaitActingChange", 2021-07-26T07:04:24.467 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.821935+0000", 2021-07-26T07:04:24.467 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.115002+0000" 2021-07-26T07:04:24.467 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.468 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.468 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.468 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.730583+0000", 2021-07-26T07:04:24.468 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.115096+0000" 2021-07-26T07:04:24.468 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.470 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.470 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.470 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:19.730531+0000", 2021-07-26T07:04:24.471 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.115120+0000" 2021-07-26T07:04:24.471 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.471 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.471 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.472 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.472 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "102", 2021-07-26T07:04:24.472 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.472 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.473 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.473 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.115121+0000", 2021-07-26T07:04:24.473 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.116334+0000" 2021-07-26T07:04:24.473 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.473 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.474 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.474 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.474 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "115", 2021-07-26T07:04:24.474 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.474 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.475 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.475 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.116358+0000", 2021-07-26T07:04:24.475 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.116412+0000" 2021-07-26T07:04:24.475 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.476 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.476 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.476 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.116458+0000", 2021-07-26T07:04:24.476 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.212769+0000" 2021-07-26T07:04:24.476 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.477 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.477 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.477 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.212771+0000", 2021-07-26T07:04:24.477 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.212935+0000" 2021-07-26T07:04:24.478 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.478 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.478 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.478 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.212936+0000", 2021-07-26T07:04:24.479 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.212976+0000" 2021-07-26T07:04:24.479 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.479 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.479 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.479 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.212976+0000", 2021-07-26T07:04:24.480 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.425440+0000" 2021-07-26T07:04:24.480 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.480 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.481 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.481 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.116435+0000", 2021-07-26T07:04:24.481 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.425461+0000" 2021-07-26T07:04:24.481 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.482 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.482 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.482 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.428972+0000", 2021-07-26T07:04:24.482 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.611713+0000" 2021-07-26T07:04:24.482 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.483 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.483 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-07-26T07:04:24.483 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.611714+0000", 2021-07-26T07:04:24.483 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.612104+0000" 2021-07-26T07:04:24.484 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.484 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.484 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-07-26T07:04:24.484 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.612104+0000", 2021-07-26T07:04:24.484 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:20.612841+0000" 2021-07-26T07:04:24.485 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.485 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.485 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovering", 2021-07-26T07:04:24.485 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.612842+0000", 2021-07-26T07:04:24.486 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:56.271519+0000" 2021-07-26T07:04:24.486 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.486 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.486 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/NotRecovering", 2021-07-26T07:04:24.486 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:56.271520+0000", 2021-07-26T07:04:24.487 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:56.271896+0000" 2021-07-26T07:04:24.487 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.487 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.487 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-07-26T07:04:24.488 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:56.271897+0000", 2021-07-26T07:04:24.488 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:57:08.815493+0000" 2021-07-26T07:04:24.488 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.488 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.488 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-07-26T07:04:24.489 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:57:08.815493+0000", 2021-07-26T07:04:24.489 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:57:08.816234+0000" 2021-07-26T07:04:24.489 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.489 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.490 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovering", 2021-07-26T07:04:24.490 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:57:08.816235+0000", 2021-07-26T07:04:24.490 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.650676+0000" 2021-07-26T07:04:24.491 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.491 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.491 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.491 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:59:34.650677+0000", 2021-07-26T07:04:24.492 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.941719+0000" 2021-07-26T07:04:24.492 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.492 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.492 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.425462+0000", 2021-07-26T07:04:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.941780+0000" 2021-07-26T07:04:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.493 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.116414+0000", 2021-07-26T07:04:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.941824+0000" 2021-07-26T07:04:24.494 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.495 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.495 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.496 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:20.116335+0000", 2021-07-26T07:04:24.496 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.941833+0000" 2021-07-26T07:04:24.496 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.497 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.497 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.498 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.498 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "115", 2021-07-26T07:04:24.499 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.499 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.499 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.500 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:59:34.941835+0000", 2021-07-26T07:04:24.500 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.942544+0000" 2021-07-26T07:04:24.501 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.501 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.502 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.502 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.503 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.503 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.504 INFO:tasks.workunit.client.0.smithi094.stdout: "pg": "3.0", 2021-07-26T07:04:24.504 INFO:tasks.workunit.client.0.smithi094.stdout: "currently": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.505 INFO:tasks.workunit.client.0.smithi094.stdout: "history": [ 2021-07-26T07:04:24.505 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.505 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "40", 2021-07-26T07:04:24.506 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.506 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.507 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.507 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.760756+0000", 2021-07-26T07:04:24.507 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:07.760776+0000" 2021-07-26T07:04:24.508 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.508 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.509 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.509 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.760789+0000", 2021-07-26T07:04:24.509 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:07.903559+0000" 2021-07-26T07:04:24.510 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.510 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.510 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.511 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.903560+0000", 2021-07-26T07:04:24.512 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:07.903782+0000" 2021-07-26T07:04:24.512 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.513 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.513 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.513 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.903783+0000", 2021-07-26T07:04:24.514 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:07.903808+0000" 2021-07-26T07:04:24.514 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.515 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.515 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.516 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.903809+0000", 2021-07-26T07:04:24.516 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.230430+0000" 2021-07-26T07:04:24.517 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.517 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.517 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.518 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.760783+0000", 2021-07-26T07:04:24.519 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.230454+0000" 2021-07-26T07:04:24.519 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.519 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.520 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.520 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.230931+0000", 2021-07-26T07:04:24.521 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.993136+0000" 2021-07-26T07:04:24.521 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.522 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.523 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.523 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.230455+0000", 2021-07-26T07:04:24.523 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.993175+0000" 2021-07-26T07:04:24.523 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.524 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.524 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.524 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.760777+0000", 2021-07-26T07:04:24.524 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.993208+0000" 2021-07-26T07:04:24.524 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.525 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.525 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.525 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:07.760750+0000", 2021-07-26T07:04:24.525 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.993215+0000" 2021-07-26T07:04:24.525 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.526 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.526 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.526 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.526 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "40", 2021-07-26T07:04:24.526 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.527 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.527 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.527 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.993216+0000", 2021-07-26T07:04:24.527 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.993608+0000" 2021-07-26T07:04:24.528 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.528 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.528 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.528 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.528 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "91", 2021-07-26T07:04:24.529 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.529 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.529 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.529 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.993616+0000", 2021-07-26T07:04:24.530 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:08.993632+0000" 2021-07-26T07:04:24.530 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.530 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.530 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.531 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.993644+0000", 2021-07-26T07:04:24.531 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:09.125609+0000" 2021-07-26T07:04:24.531 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.531 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.532 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.532 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:09.125610+0000", 2021-07-26T07:04:24.533 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:09.125760+0000" 2021-07-26T07:04:24.533 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.533 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.533 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.535 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:09.125761+0000", 2021-07-26T07:04:24.535 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:09.125787+0000" 2021-07-26T07:04:24.536 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.536 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.536 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.536 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:09.125787+0000", 2021-07-26T07:04:24.537 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:09.747850+0000" 2021-07-26T07:04:24.537 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.538 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.538 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.538 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.993638+0000", 2021-07-26T07:04:24.539 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:09.747871+0000" 2021-07-26T07:04:24.539 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.540 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.540 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.540 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:09.748251+0000", 2021-07-26T07:04:24.541 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:10.779739+0000" 2021-07-26T07:04:24.541 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.541 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.542 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.542 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:10.779740+0000", 2021-07-26T07:04:24.542 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:10.779854+0000" 2021-07-26T07:04:24.543 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.543 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.543 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.543 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:10.779855+0000", 2021-07-26T07:04:24.544 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.238037+0000" 2021-07-26T07:04:24.544 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.544 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.545 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.545 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:09.747873+0000", 2021-07-26T07:04:24.545 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.238094+0000" 2021-07-26T07:04:24.546 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.546 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.546 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.547 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.993633+0000", 2021-07-26T07:04:24.547 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.238134+0000" 2021-07-26T07:04:24.547 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.547 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.548 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.548 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:08.993609+0000", 2021-07-26T07:04:24.548 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.238141+0000" 2021-07-26T07:04:24.549 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.549 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.549 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.550 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.550 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "92", 2021-07-26T07:04:24.551 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.551 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.551 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.552 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.238141+0000", 2021-07-26T07:04:24.552 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.238850+0000" 2021-07-26T07:04:24.552 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.553 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.553 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.553 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.554 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "107", 2021-07-26T07:04:24.554 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.554 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.555 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.555 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.238859+0000", 2021-07-26T07:04:24.555 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.238896+0000" 2021-07-26T07:04:24.556 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.556 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.556 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.556 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.238919+0000", 2021-07-26T07:04:24.557 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.345054+0000" 2021-07-26T07:04:24.558 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.558 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.558 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.558 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.345055+0000", 2021-07-26T07:04:24.559 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.345180+0000" 2021-07-26T07:04:24.559 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.559 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.559 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.560 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.345181+0000", 2021-07-26T07:04:24.560 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.345203+0000" 2021-07-26T07:04:24.560 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.560 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.560 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.561 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.345204+0000", 2021-07-26T07:04:24.561 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.746492+0000" 2021-07-26T07:04:24.561 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.561 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.562 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.563 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.238909+0000", 2021-07-26T07:04:24.563 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.746507+0000" 2021-07-26T07:04:24.564 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.565 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.565 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.565 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.746649+0000", 2021-07-26T07:04:24.566 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.833129+0000" 2021-07-26T07:04:24.566 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.566 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.567 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.567 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.833130+0000", 2021-07-26T07:04:24.567 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:49.833208+0000" 2021-07-26T07:04:24.567 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.568 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.568 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.568 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.833209+0000", 2021-07-26T07:04:24.568 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.986525+0000" 2021-07-26T07:04:24.568 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.569 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.569 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.569 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.746510+0000", 2021-07-26T07:04:24.569 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.986587+0000" 2021-07-26T07:04:24.569 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.570 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.570 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.570 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.238897+0000", 2021-07-26T07:04:24.570 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.986804+0000" 2021-07-26T07:04:24.570 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.571 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.571 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.571 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:49.238850+0000", 2021-07-26T07:04:24.571 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.986811+0000" 2021-07-26T07:04:24.572 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.572 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.572 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.572 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.572 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "107", 2021-07-26T07:04:24.573 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.573 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.573 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.573 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.986813+0000", 2021-07-26T07:04:24.573 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.987524+0000" 2021-07-26T07:04:24.574 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.574 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.574 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.574 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.574 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "109", 2021-07-26T07:04:24.575 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.575 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.575 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.575 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.987533+0000", 2021-07-26T07:04:24.575 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.987565+0000" 2021-07-26T07:04:24.576 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.576 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.577 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.577 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.987585+0000", 2021-07-26T07:04:24.577 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.989065+0000" 2021-07-26T07:04:24.577 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.578 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.578 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.578 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.989066+0000", 2021-07-26T07:04:24.578 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.989272+0000" 2021-07-26T07:04:24.578 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.579 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.579 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.579 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.987574+0000", 2021-07-26T07:04:24.579 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:41.989308+0000" 2021-07-26T07:04:24.580 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.580 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.580 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/WaitActingChange", 2021-07-26T07:04:24.580 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.989308+0000", 2021-07-26T07:04:24.580 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.401390+0000" 2021-07-26T07:04:24.581 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.581 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.581 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.581 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.987566+0000", 2021-07-26T07:04:24.582 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.401438+0000" 2021-07-26T07:04:24.582 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.583 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.583 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.584 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:41.987525+0000", 2021-07-26T07:04:24.584 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.401447+0000" 2021-07-26T07:04:24.584 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.585 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.585 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.586 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.586 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "109", 2021-07-26T07:04:24.587 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.588 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.588 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.589 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.401448+0000", 2021-07-26T07:04:24.589 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.401952+0000" 2021-07-26T07:04:24.590 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.590 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.591 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.591 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.591 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "119", 2021-07-26T07:04:24.592 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.592 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.593 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.593 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.401961+0000", 2021-07-26T07:04:24.594 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.401985+0000" 2021-07-26T07:04:24.594 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.594 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.595 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.595 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.402000+0000", 2021-07-26T07:04:24.596 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.403333+0000" 2021-07-26T07:04:24.596 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.596 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.597 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.598 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.403334+0000", 2021-07-26T07:04:24.598 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.403487+0000" 2021-07-26T07:04:24.599 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.599 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.600 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.600 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.403488+0000", 2021-07-26T07:04:24.601 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.403522+0000" 2021-07-26T07:04:24.601 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.601 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.602 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.602 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.403523+0000", 2021-07-26T07:04:24.603 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.694285+0000" 2021-07-26T07:04:24.603 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.603 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.604 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.604 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.401993+0000", 2021-07-26T07:04:24.604 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:42.694310+0000" 2021-07-26T07:04:24.604 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.604 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.605 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.605 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.697641+0000", 2021-07-26T07:04:24.605 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:43.423027+0000" 2021-07-26T07:04:24.605 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.606 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.606 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-07-26T07:04:24.606 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:43.423028+0000", 2021-07-26T07:04:24.606 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:56.271779+0000" 2021-07-26T07:04:24.607 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.607 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.607 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-07-26T07:04:24.609 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:56.271780+0000", 2021-07-26T07:04:24.609 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:56.272363+0000" 2021-07-26T07:04:24.609 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.610 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.610 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovering", 2021-07-26T07:04:24.611 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:56.272364+0000", 2021-07-26T07:04:24.611 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:57:08.815201+0000" 2021-07-26T07:04:24.612 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.612 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.612 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/NotRecovering", 2021-07-26T07:04:24.613 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:57:08.815202+0000", 2021-07-26T07:04:24.613 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:57:08.815582+0000" 2021-07-26T07:04:24.614 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.614 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.615 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-07-26T07:04:24.615 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:57:08.815583+0000", 2021-07-26T07:04:24.615 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:55.981021+0000" 2021-07-26T07:04:24.616 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.617 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.617 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-07-26T07:04:24.617 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T07:01:55.981022+0000", 2021-07-26T07:04:24.618 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:55.981675+0000" 2021-07-26T07:04:24.618 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.620 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.620 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovering", 2021-07-26T07:04:24.621 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T07:01:55.981676+0000", 2021-07-26T07:04:24.621 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:04:18.692430+0000" 2021-07-26T07:04:24.622 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.622 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.623 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.623 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T07:04:18.692430+0000", 2021-07-26T07:04:24.624 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:04:18.910090+0000" 2021-07-26T07:04:24.624 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.624 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.625 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.625 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.694314+0000", 2021-07-26T07:04:24.626 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:04:18.910131+0000" 2021-07-26T07:04:24.626 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.627 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.627 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.628 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.401985+0000", 2021-07-26T07:04:24.628 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:04:18.910179+0000" 2021-07-26T07:04:24.628 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.629 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.630 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.630 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:42.401953+0000", 2021-07-26T07:04:24.630 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:04:18.910195+0000" 2021-07-26T07:04:24.631 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.631 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.631 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.631 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.632 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "119", 2021-07-26T07:04:24.632 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.632 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.632 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.632 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T07:04:18.910197+0000", 2021-07-26T07:04:24.633 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:04:18.910964+0000" 2021-07-26T07:04:24.633 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.633 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.633 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.633 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.634 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.634 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.634 INFO:tasks.workunit.client.0.smithi094.stdout: "pg": "1.0", 2021-07-26T07:04:24.634 INFO:tasks.workunit.client.0.smithi094.stdout: "currently": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.635 INFO:tasks.workunit.client.0.smithi094.stdout: "history": [ 2021-07-26T07:04:24.635 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.635 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "27", 2021-07-26T07:04:24.635 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.635 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.636 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Initial", 2021-07-26T07:04:24.636 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.706390+0000", 2021-07-26T07:04:24.636 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:01.706762+0000" 2021-07-26T07:04:24.636 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.636 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.637 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.637 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.637 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "27", 2021-07-26T07:04:24.637 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.637 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.638 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.638 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.706764+0000", 2021-07-26T07:04:24.638 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:01.706837+0000" 2021-07-26T07:04:24.638 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.639 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.639 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.639 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.639 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "30", 2021-07-26T07:04:24.639 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.640 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.640 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.640 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.706844+0000", 2021-07-26T07:04:24.640 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:01.706864+0000" 2021-07-26T07:04:24.641 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.641 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.641 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.641 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.706878+0000", 2021-07-26T07:04:24.641 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:01.832402+0000" 2021-07-26T07:04:24.642 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.642 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.642 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.642 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.832403+0000", 2021-07-26T07:04:24.642 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:01.832572+0000" 2021-07-26T07:04:24.643 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.643 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.643 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.643 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.832573+0000", 2021-07-26T07:04:24.644 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:01.832596+0000" 2021-07-26T07:04:24.644 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.644 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.644 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.644 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.832597+0000", 2021-07-26T07:04:24.645 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:02.049234+0000" 2021-07-26T07:04:24.645 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.645 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.645 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.645 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.706871+0000", 2021-07-26T07:04:24.646 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:02.049250+0000" 2021-07-26T07:04:24.646 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.646 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.646 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.646 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:02.049621+0000", 2021-07-26T07:04:24.647 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:02.529757+0000" 2021-07-26T07:04:24.647 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.647 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.647 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.647 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:02.529758+0000", 2021-07-26T07:04:24.648 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:02.530019+0000" 2021-07-26T07:04:24.648 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.648 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.648 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.649 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:02.530020+0000", 2021-07-26T07:04:24.649 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.147929+0000" 2021-07-26T07:04:24.649 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.649 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.649 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.650 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:02.049252+0000", 2021-07-26T07:04:24.650 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.148010+0000" 2021-07-26T07:04:24.650 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.650 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.650 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.651 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.706865+0000", 2021-07-26T07:04:24.651 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.148104+0000" 2021-07-26T07:04:24.651 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.651 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.651 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.652 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:01.706838+0000", 2021-07-26T07:04:24.652 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.148129+0000" 2021-07-26T07:04:24.652 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.652 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.653 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.653 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.653 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "30", 2021-07-26T07:04:24.653 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.653 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.654 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.654 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.148131+0000", 2021-07-26T07:04:24.654 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.149349+0000" 2021-07-26T07:04:24.654 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.654 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.655 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.655 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.655 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "104", 2021-07-26T07:04:24.655 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.655 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.656 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.656 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.149368+0000", 2021-07-26T07:04:24.656 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.149432+0000" 2021-07-26T07:04:24.656 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.657 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.657 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.658 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.149467+0000", 2021-07-26T07:04:24.658 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.266444+0000" 2021-07-26T07:04:24.658 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.658 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.659 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.659 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.266445+0000", 2021-07-26T07:04:24.659 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.266595+0000" 2021-07-26T07:04:24.659 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.660 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.660 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.660 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.266596+0000", 2021-07-26T07:04:24.660 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.266624+0000" 2021-07-26T07:04:24.660 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.661 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.661 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.661 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.266625+0000", 2021-07-26T07:04:24.661 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.902428+0000" 2021-07-26T07:04:24.662 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.662 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.662 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.662 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.149450+0000", 2021-07-26T07:04:24.662 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:03.902464+0000" 2021-07-26T07:04:24.663 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.663 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.663 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.663 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.902988+0000", 2021-07-26T07:04:24.663 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:04.378007+0000" 2021-07-26T07:04:24.664 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.664 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.664 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.664 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:04.378009+0000", 2021-07-26T07:04:24.664 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:49:04.378197+0000" 2021-07-26T07:04:24.665 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.665 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.665 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Clean", 2021-07-26T07:04:24.665 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:04.378198+0000", 2021-07-26T07:04:24.666 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.199958+0000" 2021-07-26T07:04:24.666 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.666 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.666 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.666 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.902466+0000", 2021-07-26T07:04:24.667 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.200018+0000" 2021-07-26T07:04:24.667 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.667 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.667 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.668 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.149433+0000", 2021-07-26T07:04:24.668 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.200284+0000" 2021-07-26T07:04:24.668 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.668 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.668 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.669 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:49:03.149350+0000", 2021-07-26T07:04:24.669 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.200293+0000" 2021-07-26T07:04:24.669 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.669 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.669 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.670 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.670 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "104", 2021-07-26T07:04:24.670 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.670 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.670 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.671 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.200294+0000", 2021-07-26T07:04:24.671 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.201190+0000" 2021-07-26T07:04:24.671 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.671 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.672 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.672 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.672 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "105", 2021-07-26T07:04:24.672 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.672 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.673 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.673 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.201208+0000", 2021-07-26T07:04:24.673 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.201245+0000" 2021-07-26T07:04:24.673 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.673 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.674 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.674 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.201269+0000", 2021-07-26T07:04:24.674 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.203114+0000" 2021-07-26T07:04:24.674 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.674 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.675 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.675 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.203115+0000", 2021-07-26T07:04:24.675 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.203280+0000" 2021-07-26T07:04:24.675 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.676 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.676 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.676 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.201255+0000", 2021-07-26T07:04:24.676 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.203297+0000" 2021-07-26T07:04:24.676 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.677 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.677 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/WaitActingChange", 2021-07-26T07:04:24.677 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.203298+0000", 2021-07-26T07:04:24.677 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.539368+0000" 2021-07-26T07:04:24.677 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.678 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.678 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.678 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.201246+0000", 2021-07-26T07:04:24.678 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.539410+0000" 2021-07-26T07:04:24.678 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.679 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.679 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.679 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.201191+0000", 2021-07-26T07:04:24.679 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.539418+0000" 2021-07-26T07:04:24.680 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.680 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.680 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.680 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.680 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "105", 2021-07-26T07:04:24.681 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.681 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.681 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.681 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.539419+0000", 2021-07-26T07:04:24.681 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.539933+0000" 2021-07-26T07:04:24.682 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.682 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.682 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.682 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.682 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "117", 2021-07-26T07:04:24.683 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.683 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.683 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Start", 2021-07-26T07:04:24.683 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.539942+0000", 2021-07-26T07:04:24.684 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.539966+0000" 2021-07-26T07:04:24.684 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.684 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.684 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-07-26T07:04:24.684 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.539982+0000", 2021-07-26T07:04:24.685 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.678319+0000" 2021-07-26T07:04:24.685 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.685 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.685 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetLog", 2021-07-26T07:04:24.685 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.678321+0000", 2021-07-26T07:04:24.686 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.678505+0000" 2021-07-26T07:04:24.686 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.686 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.686 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-07-26T07:04:24.686 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.678505+0000", 2021-07-26T07:04:24.687 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.678540+0000" 2021-07-26T07:04:24.687 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.687 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.687 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-07-26T07:04:24.687 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.678541+0000", 2021-07-26T07:04:24.688 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.912993+0000" 2021-07-26T07:04:24.688 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.688 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.688 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Peering", 2021-07-26T07:04:24.689 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.539974+0000", 2021-07-26T07:04:24.689 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:35.913014+0000" 2021-07-26T07:04:24.689 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.689 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.689 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Activating", 2021-07-26T07:04:24.690 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.916363+0000", 2021-07-26T07:04:24.690 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:56:36.383945+0000" 2021-07-26T07:04:24.690 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.690 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.690 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-07-26T07:04:24.691 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:36.383947+0000", 2021-07-26T07:04:24.691 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.651022+0000" 2021-07-26T07:04:24.691 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.691 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.692 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-07-26T07:04:24.692 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:59:34.651023+0000", 2021-07-26T07:04:24.692 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T06:59:34.651820+0000" 2021-07-26T07:04:24.692 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.693 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.693 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovering", 2021-07-26T07:04:24.693 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:59:34.651822+0000", 2021-07-26T07:04:24.693 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:55.980531+0000" 2021-07-26T07:04:24.694 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.694 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.694 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active/Recovered", 2021-07-26T07:04:24.694 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T07:01:55.980533+0000", 2021-07-26T07:04:24.694 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:56.264592+0000" 2021-07-26T07:04:24.695 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.695 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.695 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary/Active", 2021-07-26T07:04:24.695 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.913018+0000", 2021-07-26T07:04:24.695 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:56.264636+0000" 2021-07-26T07:04:24.696 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.696 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.696 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started/Primary", 2021-07-26T07:04:24.696 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.539967+0000", 2021-07-26T07:04:24.696 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:56.264675+0000" 2021-07-26T07:04:24.697 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.697 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.697 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Started", 2021-07-26T07:04:24.697 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T06:56:35.539934+0000", 2021-07-26T07:04:24.698 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:56.264682+0000" 2021-07-26T07:04:24.698 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.698 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.698 INFO:tasks.workunit.client.0.smithi094.stdout: }, 2021-07-26T07:04:24.698 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.699 INFO:tasks.workunit.client.0.smithi094.stdout: "epoch": "117", 2021-07-26T07:04:24.699 INFO:tasks.workunit.client.0.smithi094.stdout: "states": [ 2021-07-26T07:04:24.699 INFO:tasks.workunit.client.0.smithi094.stdout: { 2021-07-26T07:04:24.699 INFO:tasks.workunit.client.0.smithi094.stdout: "state": "Reset", 2021-07-26T07:04:24.699 INFO:tasks.workunit.client.0.smithi094.stdout: "enter": "2021-07-26T07:01:56.264683+0000", 2021-07-26T07:04:24.700 INFO:tasks.workunit.client.0.smithi094.stdout: "exit": "2021-07-26T07:01:56.265334+0000" 2021-07-26T07:04:24.700 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.700 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.700 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.700 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.701 INFO:tasks.workunit.client.0.smithi094.stdout: } 2021-07-26T07:04:24.701 INFO:tasks.workunit.client.0.smithi094.stdout: ] 2021-07-26T07:04:24.701 INFO:tasks.workunit.client.0.smithi094.stdout:} 2021-07-26T07:04:24.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:299: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2021-07-26T07:04:24.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:303: TEST_recovery_priority: echo TEST PASSED 2021-07-26T07:04:24.703 INFO:tasks.workunit.client.0.smithi094.stdout:TEST PASSED 2021-07-26T07:04:24.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: delete_pool test1 2021-07-26T07:04:24.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2021-07-26T07:04:24.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-07-26T07:04:24.908 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test1' does not exist 2021-07-26T07:04:24.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:307: TEST_recovery_priority: delete_pool test3 2021-07-26T07:04:24.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2021-07-26T07:04:24.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-07-26T07:04:25.448 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test3' does not exist 2021-07-26T07:04:25.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:308: TEST_recovery_priority: delete_pool test7 2021-07-26T07:04:25.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2021-07-26T07:04:25.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2021-07-26T07:04:26.177 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test7' does not exist 2021-07-26T07:04:26.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2021-07-26T07:04:26.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:26.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:26.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:26.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:26.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:26.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:26.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: return 0 2021-07-26T07:04:26.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2021-07-26T07:04:26.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-07-26T07:04:26.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:04:26.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-07-26T07:04:26.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:26.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:26.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:26.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:26.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:26.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:26.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:04:26.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:04:26.524 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:04:26.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:04:26.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:04:26.526 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:04:26.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:04:26.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:04:26.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:04:26.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:04:26.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:04:26.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:04:26.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-07-26T07:04:28.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:04:28.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:28.124 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T07:04:28.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175221 2021-07-26T07:04:28.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:04:28.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:04:28.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T07:04:28.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-recovery-prio 0 2021-07-26T07:04:28.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-07-26T07:04:28.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T07:04:28.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-07-26T07:04:28.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:28.128 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:28.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:28.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:28.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:28.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:28.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:04:28.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:04:28.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:04:28.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:04:28.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:04:28.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:04:28.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:04:28.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:04:28.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:04:28.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:04:28.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:04:28.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T07:04:28.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-07-26T07:04:28.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:04:28.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:28.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.175221 2021-07-26T07:04:28.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.175221 2021-07-26T07:04:28.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:04:28.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:04:28.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T07:04:28.144 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T07:04:28.145 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T07:04:28.158 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2021-07-26T07:04:28.159 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2021-07-26T07:04:28.217 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T07:04:28.231 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T07:04:28.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T07:04:28.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T07:04:28.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T07:04:28.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T07:04:28.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T07:04:28.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T07:04:28.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T07:04:28.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T07:04:28.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-recovery-space 2021-07-26T07:04:28.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2021-07-26T07:04:28.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2021-07-26T07:04:28.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2021-07-26T07:04:28.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2021-07-26T07:04:28.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2021-07-26T07:04:28.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2021-07-26T07:04:28.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=dd282d34-c679-466a-847c-a1573d7415d6 --auth-supported=none ' 2021-07-26T07:04:28.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2021-07-26T07:04:28.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2021-07-26T07:04:28.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: export objects=600 2021-07-26T07:04:28.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: objects=600 2021-07-26T07:04:28.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export poolprefix=test 2021-07-26T07:04:28.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: poolprefix=test 2021-07-26T07:04:28.237 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: set 2021-07-26T07:04:28.237 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T07:04:28.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: local funcs=TEST_recovery_test_simple 2021-07-26T07:04:28.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: for func in $funcs 2021-07-26T07:04:28.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: setup td/osd-recovery-space 2021-07-26T07:04:28.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2021-07-26T07:04:28.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2021-07-26T07:04:28.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2021-07-26T07:04:28.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:04:28.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2021-07-26T07:04:28.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:04:28.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:04:28.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:04:28.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:04:28.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:04:28.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:04:28.243 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:04:28.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:04:28.243 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:04:28.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:04:28.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:04:28.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:04:28.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:04:28.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:04:28.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:04:28.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:04:28.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:04:28.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:04:28.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2021-07-26T07:04:28.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:04:28.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:28.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:28.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.241905 2021-07-26T07:04:28.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:04:28.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:04:28.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2021-07-26T07:04:28.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:04:28.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:28.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:28.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.241905 2021-07-26T07:04:28.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:04:28.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T07:04:28.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T07:04:28.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:04:28.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2021-07-26T07:04:28.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: TEST_recovery_test_simple td/osd-recovery-space 2021-07-26T07:04:28.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:88: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2021-07-26T07:04:28.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local pools=1 2021-07-26T07:04:28.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local OSDS=2 2021-07-26T07:04:28.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:92: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2021-07-26T07:04:28.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2021-07-26T07:04:28.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:04:28.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:04:28.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:04:28.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2021-07-26T07:04:28.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2021-07-26T07:04:28.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:04:28.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:28.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:28.656 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:28.656 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:28.656 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:28.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.241905/$cluster-$name.asok' 2021-07-26T07:04:28.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:04:28.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:04:28.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:04:28.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:04:28.937 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:04:28.937 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:04:28.938 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:04:28.938 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:04:28.938 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:04:28.939 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:04:28.939 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:04:28.939 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:28.939 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:28.940 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.241905/ceph-mon.a.asok 2021-07-26T07:04:28.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:04:28.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.241905/ceph-mon.a.asok config get fsid 2021-07-26T07:04:29.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:04:29.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:04:29.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:04:29.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:04:29.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:04:29.034 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:04:29.034 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:04:29.034 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:04:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:04:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:29.035 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.241905/ceph-mon.a.asok 2021-07-26T07:04:29.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:04:29.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.241905/ceph-mon.a.asok config get mon_host 2021-07-26T07:04:29.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2021-07-26T07:04:29.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2021-07-26T07:04:29.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:04:29.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:04:29.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:04:29.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2021-07-26T07:04:29.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:04:30.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:04:30.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:30.004 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:30.005 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:30.005 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:30.005 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:30.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.241905/$cluster-$name.asok' 2021-07-26T07:04:30.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:04:30.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:04:30.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: export CEPH_ARGS 2021-07-26T07:04:30.037 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: expr 2 - 1 2021-07-26T07:04:30.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: seq 0 1 2021-07-26T07:04:30.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:04:30.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2021-07-26T07:04:30.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2021-07-26T07:04:30.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:04:30.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:04:30.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:04:30.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2021-07-26T07:04:30.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dd282d34-c679-466a-847c-a1573d7415d6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 ' 2021-07-26T07:04:30.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:30.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:30.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:30.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2021-07-26T07:04:30.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2021-07-26T07:04:30.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:30.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:30.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:04:30.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2021-07-26T07:04:30.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:04:30.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:30.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:30.046 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:30.046 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:30.047 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:30.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.241905/$cluster-$name.asok' 2021-07-26T07:04:30.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' 2021-07-26T07:04:30.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:30.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:04:30.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:04:30.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2021-07-26T07:04:30.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2021-07-26T07:04:30.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:30.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:30.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:30.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:04:30.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:04:30.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2021-07-26T07:04:30.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:04:30.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e51472ff-f839-4157-be87-8330aa0336d9 2021-07-26T07:04:30.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e51472ff-f839-4157-be87-8330aa0336d9' 2021-07-26T07:04:30.051 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 e51472ff-f839-4157-be87-8330aa0336d9 2021-07-26T07:04:30.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:04:30.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+Xv5gdkgcAxAApcQmT9egIrOETcyNjpsh/w== 2021-07-26T07:04:30.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+Xv5gdkgcAxAApcQmT9egIrOETcyNjpsh/w=="}' 2021-07-26T07:04:30.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e51472ff-f839-4157-be87-8330aa0336d9 -i td/osd-recovery-space/0/new.json 2021-07-26T07:04:30.412 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:04:30.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2021-07-26T07:04:30.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dd282d34-c679-466a-847c-a1573d7415d6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB+Xv5gdkgcAxAApcQmT9egIrOETcyNjpsh/w== --osd-uuid e51472ff-f839-4157-be87-8330aa0336d9 2021-07-26T07:04:30.458 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:30.444+0000 7f26cdcbcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:30.486 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:30.472+0000 7f26cdcbcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:30.489 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:30.476+0000 7f26cdcbcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:30.489 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:30.476+0000 7f26cdcbcf40 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2021-07-26T07:04:30.489 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:30.476+0000 7f26cdcbcf40 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2021-07-26T07:04:30.489 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:30.476+0000 7f26cdcbcf40 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2021-07-26T07:04:30.586 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:30.572+0000 7f26cdcbcf40 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2021-07-26T07:04:36.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2021-07-26T07:04:36.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:04:36.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:04:36.252 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:04:36.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:04:36.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:04:36.728 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:04:36.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:04:36.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dd282d34-c679-466a-847c-a1573d7415d6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:36.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:04:36.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:04:36.758 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:36.744+0000 7ff77191af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:36.759 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:36.744+0000 7ff77191af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:36.761 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:36.748+0000 7ff77191af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:37.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:04:37.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:37.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:04:37.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:37.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:37.072 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:04:37.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:37.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:37.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:37.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:37.310 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:37.296+0000 7ff77191af40 -1 Falling back to public interface 2021-07-26T07:04:37.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:38.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:38.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:38.449 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:04:38.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:38.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:38.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:38.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:39.732 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:39.720+0000 7ff77191af40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:04:39.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:39.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:39.816 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:04:39.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:39.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:39.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:40.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:41.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:41.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:41.369 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:04:41.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:04:41.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:41.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:04:41.726 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/243011,v1:127.0.0.1:6803/243011] [v2:127.0.0.1:6804/243011,v1:127.0.0.1:6805/243011] exists,up e51472ff-f839-4157-be87-8330aa0336d9 2021-07-26T07:04:41.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:04:41.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:04:41.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:04:41.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:04:41.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2021-07-26T07:04:41.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2021-07-26T07:04:41.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:04:41.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:04:41.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:04:41.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2021-07-26T07:04:41.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dd282d34-c679-466a-847c-a1573d7415d6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 ' 2021-07-26T07:04:41.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:04:41.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:04:41.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:04:41.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2021-07-26T07:04:41.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2021-07-26T07:04:41.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:04:41.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:04:41.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:04:41.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2021-07-26T07:04:41.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:04:41.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:04:41.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:04:41.738 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:04:41.739 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:04:41.739 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:04:41.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.241905/$cluster-$name.asok' 2021-07-26T07:04:41.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' 2021-07-26T07:04:41.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:04:41.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:04:41.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:04:41.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2021-07-26T07:04:41.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2021-07-26T07:04:41.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:04:41.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:04:41.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:41.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:04:41.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:04:41.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2021-07-26T07:04:41.743 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:04:41.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72a87cfe-ec01-4eaf-bc7d-f164725f214e 2021-07-26T07:04:41.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 72a87cfe-ec01-4eaf-bc7d-f164725f214e' 2021-07-26T07:04:41.744 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 72a87cfe-ec01-4eaf-bc7d-f164725f214e 2021-07-26T07:04:41.745 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:04:41.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJXv5ghlDuKxAApb76msERv+5I9B7hkMI83A== 2021-07-26T07:04:41.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJXv5ghlDuKxAApb76msERv+5I9B7hkMI83A=="}' 2021-07-26T07:04:41.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72a87cfe-ec01-4eaf-bc7d-f164725f214e -i td/osd-recovery-space/1/new.json 2021-07-26T07:04:42.270 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:04:42.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2021-07-26T07:04:42.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=dd282d34-c679-466a-847c-a1573d7415d6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCJXv5ghlDuKxAApb76msERv+5I9B7hkMI83A== --osd-uuid 72a87cfe-ec01-4eaf-bc7d-f164725f214e 2021-07-26T07:04:42.315 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:42.304+0000 7f5fa7a81f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:42.330 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:42.320+0000 7f5fa7a81f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:42.332 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:42.324+0000 7f5fa7a81f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:42.333 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:42.324+0000 7f5fa7a81f40 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2021-07-26T07:04:42.333 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:42.324+0000 7f5fa7a81f40 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2021-07-26T07:04:42.333 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:42.324+0000 7f5fa7a81f40 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2021-07-26T07:04:42.421 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:42.412+0000 7f5fa7a81f40 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2021-07-26T07:04:48.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2021-07-26T07:04:48.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:04:48.303 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:04:48.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:04:48.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:04:48.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:04:48.803 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:04:48.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=dd282d34-c679-466a-847c-a1573d7415d6 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.241905/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:04:48.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:04:48.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:04:48.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:04:48.833 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:48.823+0000 7f74d7279f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:48.835 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:48.823+0000 7f74d7279f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:48.837 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:48.827+0000 7f74d7279f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:04:49.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:04:49.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:04:49.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:04:49.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:04:49.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:04:49.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:49.237 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:04:49.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:04:49.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:49.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:49.399 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:49.387+0000 7f74d7279f40 -1 Falling back to public interface 2021-07-26T07:04:49.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:50.640 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:50.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:50.641 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:04:50.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:04:50.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:50.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:51.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:51.966 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:04:51.955+0000 7f74d7279f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:04:52.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:52.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:52.004 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:04:52.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:04:52.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:52.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:52.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:04:53.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:04:53.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:04:53.584 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:04:53.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:04:53.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:04:53.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:04:53.949 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/244417,v1:127.0.0.1:6811/244417] [v2:127.0.0.1:6812/244417,v1:127.0.0.1:6813/244417] exists,up 72a87cfe-ec01-4eaf-bc7d-f164725f214e 2021-07-26T07:04:53.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:04:53.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:04:53.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:04:53.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2021-07-26T07:04:54.766 INFO:tasks.workunit.client.0.smithi094.stderr:osd set-nearfull-ratio 0.4 2021-07-26T07:04:54.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2021-07-26T07:04:55.552 INFO:tasks.workunit.client.0.smithi094.stderr:osd set-backfillfull-ratio 0.45 2021-07-26T07:04:55.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2021-07-26T07:04:56.407 INFO:tasks.workunit.client.0.smithi094.stderr:osd set-full-ratio 0.5 2021-07-26T07:04:56.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:105: TEST_recovery_test_simple: seq 1 1 2021-07-26T07:04:56.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:105: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2021-07-26T07:04:56.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:107: TEST_recovery_test_simple: create_pool test1 1 1 2021-07-26T07:04:56.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-07-26T07:04:56.943 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test1' already exists 2021-07-26T07:04:56.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:04:57.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-07-26T07:04:58.875 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 1 2021-07-26T07:04:58.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:111: TEST_recovery_test_simple: wait_for_clean 2021-07-26T07:04:58.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:04:58.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:04:58.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:04:58.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:04:58.890 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:04:58.890 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:04:58.890 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:04:58.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:04:58.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:04:58.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:04:59.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:04:59.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:04:59.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:04:59.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:04:59.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:04:59.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:04:59.470 INFO:tasks.workunit.client.0.smithi094.stderr:1' 2021-07-26T07:04:59.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:04:59.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:59.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:04:59.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836487 2021-07-26T07:04:59.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836487 2021-07-26T07:04:59.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487' 2021-07-26T07:04:59.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:04:59.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:04:59.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672964 2021-07-26T07:04:59.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672964 2021-07-26T07:04:59.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2021-07-26T07:04:59.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:04:59.956 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836487 2021-07-26T07:04:59.956 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:04:59.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:04:59.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836487 2021-07-26T07:04:59.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:04:59.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836487 2021-07-26T07:04:59.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-07-26T07:04:59.962 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836487 2021-07-26T07:04:59.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:05:00.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836487 -lt 21474836487 2021-07-26T07:05:00.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:05:00.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672964 2021-07-26T07:05:00.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:05:00.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:05:00.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672964 2021-07-26T07:05:00.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:05:00.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672964 2021-07-26T07:05:00.323 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672964 2021-07-26T07:05:00.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-07-26T07:05:00.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:05:00.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672964 -lt 42949672964 2021-07-26T07:05:00.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:05:00.676 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:00.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:05:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:05:01.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:05:01.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:05:01.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:05:01.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:05:01.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:05:01.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:05:01.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:05:01.492 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:05:01.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:05:01.493 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:05:01.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:05:01.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:05:01.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:05:01.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:113: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2021-07-26T07:05:01.946 INFO:tasks.workunit.client.0.smithi094.stderr:5+0 records in 2021-07-26T07:05:01.946 INFO:tasks.workunit.client.0.smithi094.stderr:5+0 records out 2021-07-26T07:05:01.946 INFO:tasks.workunit.client.0.smithi094.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 0.000184133 s, 27.8 MB/s 2021-07-26T07:05:01.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: seq 1 600 2021-07-26T07:05:01.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:01.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2021-07-26T07:05:02.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2021-07-26T07:05:02.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2021-07-26T07:05:02.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2021-07-26T07:05:02.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2021-07-26T07:05:02.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2021-07-26T07:05:02.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2021-07-26T07:05:02.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2021-07-26T07:05:02.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2021-07-26T07:05:02.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:02.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2021-07-26T07:05:03.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2021-07-26T07:05:03.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2021-07-26T07:05:03.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2021-07-26T07:05:03.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2021-07-26T07:05:03.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2021-07-26T07:05:03.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2021-07-26T07:05:03.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2021-07-26T07:05:03.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2021-07-26T07:05:03.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2021-07-26T07:05:03.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2021-07-26T07:05:03.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:03.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2021-07-26T07:05:04.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2021-07-26T07:05:04.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2021-07-26T07:05:04.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2021-07-26T07:05:04.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2021-07-26T07:05:04.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2021-07-26T07:05:04.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2021-07-26T07:05:04.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2021-07-26T07:05:04.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:04.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2021-07-26T07:05:05.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2021-07-26T07:05:05.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2021-07-26T07:05:05.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2021-07-26T07:05:05.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2021-07-26T07:05:05.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2021-07-26T07:05:05.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2021-07-26T07:05:05.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2021-07-26T07:05:05.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2021-07-26T07:05:05.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2021-07-26T07:05:05.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:05.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2021-07-26T07:05:06.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2021-07-26T07:05:06.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2021-07-26T07:05:06.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2021-07-26T07:05:06.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2021-07-26T07:05:06.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2021-07-26T07:05:06.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2021-07-26T07:05:06.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2021-07-26T07:05:06.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2021-07-26T07:05:06.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2021-07-26T07:05:06.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:06.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2021-07-26T07:05:07.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2021-07-26T07:05:07.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2021-07-26T07:05:07.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2021-07-26T07:05:07.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2021-07-26T07:05:07.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2021-07-26T07:05:07.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2021-07-26T07:05:07.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2021-07-26T07:05:07.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2021-07-26T07:05:07.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2021-07-26T07:05:07.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:07.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2021-07-26T07:05:08.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2021-07-26T07:05:08.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2021-07-26T07:05:08.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2021-07-26T07:05:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2021-07-26T07:05:08.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2021-07-26T07:05:08.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2021-07-26T07:05:08.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2021-07-26T07:05:08.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2021-07-26T07:05:08.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2021-07-26T07:05:08.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:08.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2021-07-26T07:05:09.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2021-07-26T07:05:09.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2021-07-26T07:05:09.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2021-07-26T07:05:09.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2021-07-26T07:05:09.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2021-07-26T07:05:09.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2021-07-26T07:05:09.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2021-07-26T07:05:09.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2021-07-26T07:05:09.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:09.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2021-07-26T07:05:10.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2021-07-26T07:05:10.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2021-07-26T07:05:10.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2021-07-26T07:05:10.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2021-07-26T07:05:10.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2021-07-26T07:05:10.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2021-07-26T07:05:10.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2021-07-26T07:05:10.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2021-07-26T07:05:10.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2021-07-26T07:05:10.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:10.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2021-07-26T07:05:11.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2021-07-26T07:05:11.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2021-07-26T07:05:11.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2021-07-26T07:05:11.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2021-07-26T07:05:11.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2021-07-26T07:05:11.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2021-07-26T07:05:11.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2021-07-26T07:05:11.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2021-07-26T07:05:11.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2021-07-26T07:05:11.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:11.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2021-07-26T07:05:12.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2021-07-26T07:05:12.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2021-07-26T07:05:12.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2021-07-26T07:05:12.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2021-07-26T07:05:12.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2021-07-26T07:05:12.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2021-07-26T07:05:12.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2021-07-26T07:05:12.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2021-07-26T07:05:12.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2021-07-26T07:05:12.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:12.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2021-07-26T07:05:13.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2021-07-26T07:05:13.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2021-07-26T07:05:13.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2021-07-26T07:05:13.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2021-07-26T07:05:13.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2021-07-26T07:05:13.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2021-07-26T07:05:13.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2021-07-26T07:05:13.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2021-07-26T07:05:13.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2021-07-26T07:05:13.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:13.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2021-07-26T07:05:14.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2021-07-26T07:05:14.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2021-07-26T07:05:14.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2021-07-26T07:05:14.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2021-07-26T07:05:14.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2021-07-26T07:05:14.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2021-07-26T07:05:14.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2021-07-26T07:05:14.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2021-07-26T07:05:14.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:14.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2021-07-26T07:05:15.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2021-07-26T07:05:15.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2021-07-26T07:05:15.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2021-07-26T07:05:15.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2021-07-26T07:05:15.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2021-07-26T07:05:15.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2021-07-26T07:05:15.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2021-07-26T07:05:15.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2021-07-26T07:05:15.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2021-07-26T07:05:15.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:15.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2021-07-26T07:05:16.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2021-07-26T07:05:16.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2021-07-26T07:05:16.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2021-07-26T07:05:16.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2021-07-26T07:05:16.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2021-07-26T07:05:16.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2021-07-26T07:05:16.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2021-07-26T07:05:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2021-07-26T07:05:16.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2021-07-26T07:05:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:16.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2021-07-26T07:05:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2021-07-26T07:05:17.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2021-07-26T07:05:17.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2021-07-26T07:05:17.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2021-07-26T07:05:17.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2021-07-26T07:05:17.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2021-07-26T07:05:17.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2021-07-26T07:05:17.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2021-07-26T07:05:17.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2021-07-26T07:05:17.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:17.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2021-07-26T07:05:18.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2021-07-26T07:05:18.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2021-07-26T07:05:18.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2021-07-26T07:05:18.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2021-07-26T07:05:18.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2021-07-26T07:05:18.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2021-07-26T07:05:18.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2021-07-26T07:05:18.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2021-07-26T07:05:18.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2021-07-26T07:05:18.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:18.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2021-07-26T07:05:19.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2021-07-26T07:05:19.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2021-07-26T07:05:19.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2021-07-26T07:05:19.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2021-07-26T07:05:19.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2021-07-26T07:05:19.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2021-07-26T07:05:19.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2021-07-26T07:05:19.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2021-07-26T07:05:19.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:19.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2021-07-26T07:05:20.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2021-07-26T07:05:20.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2021-07-26T07:05:20.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2021-07-26T07:05:20.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2021-07-26T07:05:20.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2021-07-26T07:05:20.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2021-07-26T07:05:20.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2021-07-26T07:05:20.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2021-07-26T07:05:20.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2021-07-26T07:05:20.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:20.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2021-07-26T07:05:21.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2021-07-26T07:05:21.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2021-07-26T07:05:21.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2021-07-26T07:05:21.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2021-07-26T07:05:21.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2021-07-26T07:05:21.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2021-07-26T07:05:21.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2021-07-26T07:05:21.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2021-07-26T07:05:21.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:21.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2021-07-26T07:05:22.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2021-07-26T07:05:22.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2021-07-26T07:05:22.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2021-07-26T07:05:22.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2021-07-26T07:05:22.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2021-07-26T07:05:22.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2021-07-26T07:05:22.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2021-07-26T07:05:22.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2021-07-26T07:05:22.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2021-07-26T07:05:22.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2021-07-26T07:05:22.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:22.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2021-07-26T07:05:23.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2021-07-26T07:05:23.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2021-07-26T07:05:23.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2021-07-26T07:05:23.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2021-07-26T07:05:23.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2021-07-26T07:05:23.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2021-07-26T07:05:23.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2021-07-26T07:05:23.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2021-07-26T07:05:23.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2021-07-26T07:05:23.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:23.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2021-07-26T07:05:24.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2021-07-26T07:05:24.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2021-07-26T07:05:24.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2021-07-26T07:05:24.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2021-07-26T07:05:24.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2021-07-26T07:05:24.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2021-07-26T07:05:24.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2021-07-26T07:05:24.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2021-07-26T07:05:24.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:24.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2021-07-26T07:05:25.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2021-07-26T07:05:25.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2021-07-26T07:05:25.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2021-07-26T07:05:25.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2021-07-26T07:05:25.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2021-07-26T07:05:25.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2021-07-26T07:05:25.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2021-07-26T07:05:25.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2021-07-26T07:05:25.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2021-07-26T07:05:25.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:25.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2021-07-26T07:05:26.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2021-07-26T07:05:26.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2021-07-26T07:05:26.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2021-07-26T07:05:26.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2021-07-26T07:05:26.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2021-07-26T07:05:26.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2021-07-26T07:05:26.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2021-07-26T07:05:26.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2021-07-26T07:05:26.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2021-07-26T07:05:26.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:26.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2021-07-26T07:05:27.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2021-07-26T07:05:27.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2021-07-26T07:05:27.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2021-07-26T07:05:27.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2021-07-26T07:05:27.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2021-07-26T07:05:27.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2021-07-26T07:05:27.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2021-07-26T07:05:27.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2021-07-26T07:05:27.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2021-07-26T07:05:27.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:27.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2021-07-26T07:05:28.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2021-07-26T07:05:28.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2021-07-26T07:05:28.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2021-07-26T07:05:28.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2021-07-26T07:05:28.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2021-07-26T07:05:28.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2021-07-26T07:05:28.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2021-07-26T07:05:28.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2021-07-26T07:05:28.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2021-07-26T07:05:28.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:28.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2021-07-26T07:05:29.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2021-07-26T07:05:29.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2021-07-26T07:05:29.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2021-07-26T07:05:29.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2021-07-26T07:05:29.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2021-07-26T07:05:29.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2021-07-26T07:05:29.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2021-07-26T07:05:29.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2021-07-26T07:05:29.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:29.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2021-07-26T07:05:30.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:30.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2021-07-26T07:05:30.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:30.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2021-07-26T07:05:30.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:30.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2021-07-26T07:05:30.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:30.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2021-07-26T07:05:30.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:30.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2021-07-26T07:05:30.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:30.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2021-07-26T07:05:30.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:30.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2021-07-26T07:05:31.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2021-07-26T07:05:31.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2021-07-26T07:05:31.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2021-07-26T07:05:31.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2021-07-26T07:05:31.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2021-07-26T07:05:31.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2021-07-26T07:05:31.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2021-07-26T07:05:31.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2021-07-26T07:05:31.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2021-07-26T07:05:31.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:31.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2021-07-26T07:05:32.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2021-07-26T07:05:32.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2021-07-26T07:05:32.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2021-07-26T07:05:32.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2021-07-26T07:05:32.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2021-07-26T07:05:32.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2021-07-26T07:05:32.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2021-07-26T07:05:32.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2021-07-26T07:05:32.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:32.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2021-07-26T07:05:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2021-07-26T07:05:33.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2021-07-26T07:05:33.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2021-07-26T07:05:33.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2021-07-26T07:05:33.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2021-07-26T07:05:33.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2021-07-26T07:05:33.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2021-07-26T07:05:33.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2021-07-26T07:05:33.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2021-07-26T07:05:33.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2021-07-26T07:05:33.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:33.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2021-07-26T07:05:34.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2021-07-26T07:05:34.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2021-07-26T07:05:34.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2021-07-26T07:05:34.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2021-07-26T07:05:34.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2021-07-26T07:05:34.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2021-07-26T07:05:34.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2021-07-26T07:05:34.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2021-07-26T07:05:34.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:34.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2021-07-26T07:05:35.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2021-07-26T07:05:35.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2021-07-26T07:05:35.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2021-07-26T07:05:35.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2021-07-26T07:05:35.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2021-07-26T07:05:35.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2021-07-26T07:05:35.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2021-07-26T07:05:35.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:35.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2021-07-26T07:05:36.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2021-07-26T07:05:36.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2021-07-26T07:05:36.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2021-07-26T07:05:36.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2021-07-26T07:05:36.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2021-07-26T07:05:36.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2021-07-26T07:05:36.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2021-07-26T07:05:36.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2021-07-26T07:05:36.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2021-07-26T07:05:36.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:36.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2021-07-26T07:05:37.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2021-07-26T07:05:37.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2021-07-26T07:05:37.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2021-07-26T07:05:37.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2021-07-26T07:05:37.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2021-07-26T07:05:37.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2021-07-26T07:05:37.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2021-07-26T07:05:37.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2021-07-26T07:05:37.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2021-07-26T07:05:37.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:37.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2021-07-26T07:05:38.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2021-07-26T07:05:38.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2021-07-26T07:05:38.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2021-07-26T07:05:38.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2021-07-26T07:05:38.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2021-07-26T07:05:38.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2021-07-26T07:05:38.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2021-07-26T07:05:38.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2021-07-26T07:05:38.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2021-07-26T07:05:38.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:38.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2021-07-26T07:05:39.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2021-07-26T07:05:39.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2021-07-26T07:05:39.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2021-07-26T07:05:39.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2021-07-26T07:05:39.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2021-07-26T07:05:39.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2021-07-26T07:05:39.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2021-07-26T07:05:39.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2021-07-26T07:05:39.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2021-07-26T07:05:39.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:39.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2021-07-26T07:05:40.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2021-07-26T07:05:40.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2021-07-26T07:05:40.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2021-07-26T07:05:40.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2021-07-26T07:05:40.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2021-07-26T07:05:40.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2021-07-26T07:05:40.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2021-07-26T07:05:40.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2021-07-26T07:05:40.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2021-07-26T07:05:40.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:40.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2021-07-26T07:05:41.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2021-07-26T07:05:41.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2021-07-26T07:05:41.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2021-07-26T07:05:41.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2021-07-26T07:05:41.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2021-07-26T07:05:41.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2021-07-26T07:05:41.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2021-07-26T07:05:41.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2021-07-26T07:05:41.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2021-07-26T07:05:41.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:41.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2021-07-26T07:05:42.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2021-07-26T07:05:42.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2021-07-26T07:05:42.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2021-07-26T07:05:42.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2021-07-26T07:05:42.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2021-07-26T07:05:42.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2021-07-26T07:05:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2021-07-26T07:05:42.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2021-07-26T07:05:42.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2021-07-26T07:05:42.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:42.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2021-07-26T07:05:43.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2021-07-26T07:05:43.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2021-07-26T07:05:43.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2021-07-26T07:05:43.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2021-07-26T07:05:43.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2021-07-26T07:05:43.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2021-07-26T07:05:43.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2021-07-26T07:05:43.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2021-07-26T07:05:43.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2021-07-26T07:05:43.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:43.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2021-07-26T07:05:44.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2021-07-26T07:05:44.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2021-07-26T07:05:44.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2021-07-26T07:05:44.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2021-07-26T07:05:44.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2021-07-26T07:05:44.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2021-07-26T07:05:44.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2021-07-26T07:05:44.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2021-07-26T07:05:44.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2021-07-26T07:05:44.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:44.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2021-07-26T07:05:45.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2021-07-26T07:05:45.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2021-07-26T07:05:45.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2021-07-26T07:05:45.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2021-07-26T07:05:45.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2021-07-26T07:05:45.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2021-07-26T07:05:45.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2021-07-26T07:05:45.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2021-07-26T07:05:45.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2021-07-26T07:05:45.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:45.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2021-07-26T07:05:46.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2021-07-26T07:05:46.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2021-07-26T07:05:46.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2021-07-26T07:05:46.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2021-07-26T07:05:46.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2021-07-26T07:05:46.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2021-07-26T07:05:46.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2021-07-26T07:05:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2021-07-26T07:05:46.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:46.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2021-07-26T07:05:47.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2021-07-26T07:05:47.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2021-07-26T07:05:47.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2021-07-26T07:05:47.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2021-07-26T07:05:47.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2021-07-26T07:05:47.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2021-07-26T07:05:47.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2021-07-26T07:05:47.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2021-07-26T07:05:47.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:47.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2021-07-26T07:05:48.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2021-07-26T07:05:48.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2021-07-26T07:05:48.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2021-07-26T07:05:48.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2021-07-26T07:05:48.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2021-07-26T07:05:48.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2021-07-26T07:05:48.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2021-07-26T07:05:48.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2021-07-26T07:05:48.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2021-07-26T07:05:48.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:48.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2021-07-26T07:05:49.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2021-07-26T07:05:49.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2021-07-26T07:05:49.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2021-07-26T07:05:49.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2021-07-26T07:05:49.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2021-07-26T07:05:49.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2021-07-26T07:05:49.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2021-07-26T07:05:49.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2021-07-26T07:05:49.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:49.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2021-07-26T07:05:50.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2021-07-26T07:05:50.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2021-07-26T07:05:50.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2021-07-26T07:05:50.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2021-07-26T07:05:50.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2021-07-26T07:05:50.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2021-07-26T07:05:50.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2021-07-26T07:05:50.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2021-07-26T07:05:50.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2021-07-26T07:05:50.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:50.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2021-07-26T07:05:51.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2021-07-26T07:05:51.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2021-07-26T07:05:51.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2021-07-26T07:05:51.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2021-07-26T07:05:51.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2021-07-26T07:05:51.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2021-07-26T07:05:51.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2021-07-26T07:05:51.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2021-07-26T07:05:51.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2021-07-26T07:05:51.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:51.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2021-07-26T07:05:52.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2021-07-26T07:05:52.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2021-07-26T07:05:52.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2021-07-26T07:05:52.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2021-07-26T07:05:52.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2021-07-26T07:05:52.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2021-07-26T07:05:52.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2021-07-26T07:05:52.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2021-07-26T07:05:52.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2021-07-26T07:05:52.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:52.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2021-07-26T07:05:53.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2021-07-26T07:05:53.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2021-07-26T07:05:53.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2021-07-26T07:05:53.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2021-07-26T07:05:53.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2021-07-26T07:05:53.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2021-07-26T07:05:53.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2021-07-26T07:05:53.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2021-07-26T07:05:53.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2021-07-26T07:05:53.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:53.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2021-07-26T07:05:54.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2021-07-26T07:05:54.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2021-07-26T07:05:54.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2021-07-26T07:05:54.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2021-07-26T07:05:54.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2021-07-26T07:05:54.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2021-07-26T07:05:54.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2021-07-26T07:05:54.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2021-07-26T07:05:54.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2021-07-26T07:05:54.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:54.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2021-07-26T07:05:55.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2021-07-26T07:05:55.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2021-07-26T07:05:55.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2021-07-26T07:05:55.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2021-07-26T07:05:55.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2021-07-26T07:05:55.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2021-07-26T07:05:55.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2021-07-26T07:05:55.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2021-07-26T07:05:55.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:55.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2021-07-26T07:05:56.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2021-07-26T07:05:56.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2021-07-26T07:05:56.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2021-07-26T07:05:56.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2021-07-26T07:05:56.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2021-07-26T07:05:56.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2021-07-26T07:05:56.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2021-07-26T07:05:56.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2021-07-26T07:05:56.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2021-07-26T07:05:56.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:56.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2021-07-26T07:05:57.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2021-07-26T07:05:57.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2021-07-26T07:05:57.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2021-07-26T07:05:57.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2021-07-26T07:05:57.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2021-07-26T07:05:57.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2021-07-26T07:05:57.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2021-07-26T07:05:57.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2021-07-26T07:05:57.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:57.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2021-07-26T07:05:58.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2021-07-26T07:05:58.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2021-07-26T07:05:58.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2021-07-26T07:05:58.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2021-07-26T07:05:58.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2021-07-26T07:05:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2021-07-26T07:05:58.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2021-07-26T07:05:58.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2021-07-26T07:05:58.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2021-07-26T07:05:58.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:58.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2021-07-26T07:05:59.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2021-07-26T07:05:59.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2021-07-26T07:05:59.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2021-07-26T07:05:59.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2021-07-26T07:05:59.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2021-07-26T07:05:59.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2021-07-26T07:05:59.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2021-07-26T07:05:59.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2021-07-26T07:05:59.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:05:59.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2021-07-26T07:06:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2021-07-26T07:06:00.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2021-07-26T07:06:00.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2021-07-26T07:06:00.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2021-07-26T07:06:00.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2021-07-26T07:06:00.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2021-07-26T07:06:00.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2021-07-26T07:06:00.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2021-07-26T07:06:00.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2021-07-26T07:06:00.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:00.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2021-07-26T07:06:01.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:01.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2021-07-26T07:06:01.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:01.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2021-07-26T07:06:01.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:01.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2021-07-26T07:06:01.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:01.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2021-07-26T07:06:01.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:01.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2021-07-26T07:06:01.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:01.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2021-07-26T07:06:01.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:01.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2021-07-26T07:06:02.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2021-07-26T07:06:02.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2021-07-26T07:06:02.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2021-07-26T07:06:02.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2021-07-26T07:06:02.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2021-07-26T07:06:02.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2021-07-26T07:06:02.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2021-07-26T07:06:02.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2021-07-26T07:06:02.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:02.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2021-07-26T07:06:03.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2021-07-26T07:06:03.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2021-07-26T07:06:03.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2021-07-26T07:06:03.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2021-07-26T07:06:03.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2021-07-26T07:06:03.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2021-07-26T07:06:03.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2021-07-26T07:06:03.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2021-07-26T07:06:03.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2021-07-26T07:06:03.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:03.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2021-07-26T07:06:04.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:04.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2021-07-26T07:06:04.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-07-26T07:06:04.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2021-07-26T07:06:04.243 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: expr 2 - 1 2021-07-26T07:06:04.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: seq 0 1 2021-07-26T07:06:04.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:06:04.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2021-07-26T07:06:04.384 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T07:06:04.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2021-07-26T07:06:04.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2021-07-26T07:06:04.521 INFO:tasks.workunit.client.0.smithi094.stdout:{} 2021-07-26T07:06:04.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:123: TEST_recovery_test_simple: sleep 5 2021-07-26T07:06:09.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:125: TEST_recovery_test_simple: ceph pg dump pgs 2021-07-26T07:06:09.877 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T07:06:09.878 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 591 0 0 0 0 3025920 0 0 591 591 active+clean 2021-07-26T07:04:59.327168+0000 23'591 23:608 [1] 1 [1] 1 0'0 2021-07-26T07:04:56.742954+0000 0'0 2021-07-26T07:04:56.742954+0000 0 2021-07-26T07:06:09.878 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T07:06:09.878 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T07:06:09.880 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T07:06:09.889 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:127: TEST_recovery_test_simple: seq 1 1 2021-07-26T07:06:09.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:127: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2021-07-26T07:06:09.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:129: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2021-07-26T07:06:10.666 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 2 2021-07-26T07:06:10.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:133: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2021-07-26T07:06:10.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:76: wait_for_recovery_toofull: local timeout=30 2021-07-26T07:06:10.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2021-07-26T07:06:10.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:51: wait_for_state: local state=recovery_toofull 2021-07-26T07:06:10.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local cur_in_state 2021-07-26T07:06:10.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: delays=($(get_timeout_delays $2 5)) 2021-07-26T07:06:10.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: get_timeout_delays 30 5 2021-07-26T07:06:10.684 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:06:10.684 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:06:10.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:06:10.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:06:10.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:06:10.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local -a delays 2021-07-26T07:06:10.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -i loop=0 2021-07-26T07:06:10.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:56: wait_for_state: flush_pg_stats 2021-07-26T07:06:10.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:06:10.739 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:06:11.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:06:11.215 INFO:tasks.workunit.client.0.smithi094.stderr:1' 2021-07-26T07:06:11.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:06:11.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:11.216 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:06:11.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836504 2021-07-26T07:06:11.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836504 2021-07-26T07:06:11.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504' 2021-07-26T07:06:11.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:06:11.356 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:06:11.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672980 2021-07-26T07:06:11.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672980 2021-07-26T07:06:11.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836504 1-42949672980' 2021-07-26T07:06:11.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:11.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836504 2021-07-26T07:06:11.503 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:11.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:06:11.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836504 2021-07-26T07:06:11.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:11.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836504 2021-07-26T07:06:11.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2021-07-26T07:06:11.508 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836504 2021-07-26T07:06:11.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:11.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836501 -lt 21474836504 2021-07-26T07:06:11.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:06:12.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:06:12.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:06:13.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836504 -lt 21474836504 2021-07-26T07:06:13.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:06:13.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672980 2021-07-26T07:06:13.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:06:13.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:06:13.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672980 2021-07-26T07:06:13.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:06:13.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672980 2021-07-26T07:06:13.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-07-26T07:06:13.259 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672980 2021-07-26T07:06:13.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:06:13.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672980 -lt 42949672980 2021-07-26T07:06:13.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: get_num_pgs 2021-07-26T07:06:13.636 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:06:13.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:06:14.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: test 1 == 0 2021-07-26T07:06:14.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:61: wait_for_state: true 2021-07-26T07:06:14.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: get_num_in_state recovery_toofull 2021-07-26T07:06:14.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:42: get_num_in_state: local state=recovery_toofull 2021-07-26T07:06:14.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local expression 2021-07-26T07:06:14.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2021-07-26T07:06:14.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: ceph --format json pg dump pgs 2021-07-26T07:06:14.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2021-07-26T07:06:14.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: cur_in_state=1 2021-07-26T07:06:14.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: test 1 -gt 0 2021-07-26T07:06:14.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: break 2021-07-26T07:06:14.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:71: wait_for_state: return 0 2021-07-26T07:06:14.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:135: TEST_recovery_test_simple: ERRORS=0 2021-07-26T07:06:14.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ceph pg dump pgs 2021-07-26T07:06:14.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: grep +recovery_toofull 2021-07-26T07:06:14.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: wc -l 2021-07-26T07:06:14.792 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T07:06:14.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2021-07-26T07:06:14.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:142: TEST_recovery_test_simple: ceph pg dump pgs 2021-07-26T07:06:15.128 INFO:tasks.workunit.client.0.smithi094.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-07-26T07:06:15.128 INFO:tasks.workunit.client.0.smithi094.stdout:1.0 600 0 600 0 0 3072000 0 0 600 600 active+recovery_toofull+undersized+degraded+remapped 2021-07-26T07:06:11.223796+0000 23'600 27:628 [1,0] 1 [1] 1 0'0 2021-07-26T07:04:56.742954+0000 0'0 2021-07-26T07:04:56.742954+0000 0 2021-07-26T07:06:15.128 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T07:06:15.128 INFO:tasks.workunit.client.0.smithi094.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-07-26T07:06:15.131 INFO:tasks.workunit.client.0.smithi094.stderr:dumped pgs 2021-07-26T07:06:15.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph status 2021-07-26T07:06:15.598 INFO:tasks.workunit.client.0.smithi094.stdout: cluster: 2021-07-26T07:06:15.599 INFO:tasks.workunit.client.0.smithi094.stdout: id: dd282d34-c679-466a-847c-a1573d7415d6 2021-07-26T07:06:15.599 INFO:tasks.workunit.client.0.smithi094.stdout: health: HEALTH_ERR 2021-07-26T07:06:15.600 INFO:tasks.workunit.client.0.smithi094.stdout: 2 full osd(s) 2021-07-26T07:06:15.600 INFO:tasks.workunit.client.0.smithi094.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2021-07-26T07:06:15.600 INFO:tasks.workunit.client.0.smithi094.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2021-07-26T07:06:15.600 INFO:tasks.workunit.client.0.smithi094.stdout: 1 pool(s) do not have an application enabled 2021-07-26T07:06:15.601 INFO:tasks.workunit.client.0.smithi094.stdout: 1 pool(s) full 2021-07-26T07:06:15.601 INFO:tasks.workunit.client.0.smithi094.stdout: OSD count 2 < osd_pool_default_size 3 2021-07-26T07:06:15.601 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T07:06:15.601 INFO:tasks.workunit.client.0.smithi094.stdout: services: 2021-07-26T07:06:15.602 INFO:tasks.workunit.client.0.smithi094.stdout: mon: 1 daemons, quorum a (age 106s) 2021-07-26T07:06:15.602 INFO:tasks.workunit.client.0.smithi094.stdout: mgr: x(active, since 101s) 2021-07-26T07:06:15.602 INFO:tasks.workunit.client.0.smithi094.stdout: osd: 2 osds: 2 up (since 82s), 2 in (since 93s); 1 remapped pgs 2021-07-26T07:06:15.602 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T07:06:15.602 INFO:tasks.workunit.client.0.smithi094.stdout: data: 2021-07-26T07:06:15.603 INFO:tasks.workunit.client.0.smithi094.stdout: pools: 1 pools, 1 pgs 2021-07-26T07:06:15.603 INFO:tasks.workunit.client.0.smithi094.stdout: objects: 600 objects, 2.9 MiB 2021-07-26T07:06:15.603 INFO:tasks.workunit.client.0.smithi094.stdout: usage: 2.9 MiB used, 4.1 MiB / 7.0 MiB avail 2021-07-26T07:06:15.603 INFO:tasks.workunit.client.0.smithi094.stdout: pgs: 600/1200 objects degraded (50.000%) 2021-07-26T07:06:15.603 INFO:tasks.workunit.client.0.smithi094.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2021-07-26T07:06:15.604 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T07:06:15.604 INFO:tasks.workunit.client.0.smithi094.stdout: io: 2021-07-26T07:06:15.604 INFO:tasks.workunit.client.0.smithi094.stdout: client: 7.5 KiB/s wr, 0 op/s rd, 1 op/s wr 2021-07-26T07:06:15.604 INFO:tasks.workunit.client.0.smithi094.stdout: 2021-07-26T07:06:15.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status --format=json-pretty 2021-07-26T07:06:16.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:146: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2021-07-26T07:06:16.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:146: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2021-07-26T07:06:16.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:146: TEST_recovery_test_simple: SEV=HEALTH_ERR 2021-07-26T07:06:16.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2021-07-26T07:06:16.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2021-07-26T07:06:16.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2021-07-26T07:06:16.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2021-07-26T07:06:16.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2021-07-26T07:06:16.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:156: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2021-07-26T07:06:16.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:158: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2021-07-26T07:06:16.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:163: TEST_recovery_test_simple: seq 1 1 2021-07-26T07:06:16.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:163: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2021-07-26T07:06:16.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:165: TEST_recovery_test_simple: delete_pool test1 2021-07-26T07:06:16.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2021-07-26T07:06:16.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-07-26T07:06:16.684 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test1' does not exist 2021-07-26T07:06:16.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:167: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2021-07-26T07:06:16.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:16.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:16.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:18.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:18.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: teardown td/osd-recovery-space 2021-07-26T07:06:18.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2021-07-26T07:06:18.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:06:18.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2021-07-26T07:06:18.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:18.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:18.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:18.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:18.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:18.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:18.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:06:18.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:06:18.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:06:18.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:06:18.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:06:18.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:06:18.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:06:18.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:06:18.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:06:18.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:06:18.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:06:18.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:06:18.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2021-07-26T07:06:18.063 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:06:18.063 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:18.064 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:06:18.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.241905 2021-07-26T07:06:18.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:06:18.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:06:18.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: code=0 2021-07-26T07:06:18.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2159: main: teardown td/osd-recovery-space 0 2021-07-26T07:06:18.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2021-07-26T07:06:18.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-07-26T07:06:18.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2021-07-26T07:06:18.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:18.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:18.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:18.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:18.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:18.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:18.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:06:18.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:06:18.073 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:06:18.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:06:18.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:06:18.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:06:18.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:06:18.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:06:18.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:06:18.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:06:18.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:06:18.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-07-26T07:06:18.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2021-07-26T07:06:18.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:06:18.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:18.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.241905 2021-07-26T07:06:18.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.241905 2021-07-26T07:06:18.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:06:18.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:06:18.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: main: return 0 2021-07-26T07:06:18.089 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-07-26T07:06:18.089 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-07-26T07:06:18.104 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2021-07-26T07:06:18.105 DEBUG:teuthology.orchestra.run.smithi094:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=197e564fdaee12596de1dc8f3253d1704ae66d5d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2021-07-26T07:06:18.167 INFO:tasks.workunit.client.0.smithi094.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-07-26T07:06:18.179 INFO:tasks.workunit.client.0.smithi094.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-07-26T07:06:18.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T07:06:18.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2021-07-26T07:06:18.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: export PYTHONWARNINGS=ignore 2021-07-26T07:06:18.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: PYTHONWARNINGS=ignore 2021-07-26T07:06:18.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: export CEPH_CONF=/dev/null 2021-07-26T07:06:18.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: CEPH_CONF=/dev/null 2021-07-26T07:06:18.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: unset CEPH_ARGS 2021-07-26T07:06:18.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: main: local code 2021-07-26T07:06:18.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: run td/osd-recovery-stats 2021-07-26T07:06:18.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2021-07-26T07:06:18.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2021-07-26T07:06:18.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2021-07-26T07:06:18.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2021-07-26T07:06:18.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2021-07-26T07:06:18.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2021-07-26T07:06:18.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none ' 2021-07-26T07:06:18.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2021-07-26T07:06:18.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:06:18.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2021-07-26T07:06:18.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2021-07-26T07:06:18.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2021-07-26T07:06:18.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2021-07-26T07:06:18.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2021-07-26T07:06:18.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2021-07-26T07:06:18.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2021-07-26T07:06:18.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-07-26T07:06:18.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2021-07-26T07:06:18.188 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_recovery_erasure_remapped 2021-07-26T07:06:18.188 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_recovery_multi 2021-07-26T07:06:18.189 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_recovery_replicated_out1 2021-07-26T07:06:18.189 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_recovery_sizedown 2021-07-26T07:06:18.189 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_recovery_sizeup 2021-07-26T07:06:18.189 INFO:tasks.workunit.client.0.smithi094.stderr:TEST_recovery_undersized' 2021-07-26T07:06:18.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-07-26T07:06:18.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-07-26T07:06:18.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-07-26T07:06:18.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-07-26T07:06:18.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:06:18.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:06:18.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:06:18.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:06:18.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:06:18.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:06:18.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:06:18.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:06:18.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:06:18.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:06:18.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:06:18.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:06:18.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:06:18.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:06:18.195 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:06:18.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:06:18.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:06:18.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:06:18.197 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:06:18.197 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:06:18.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:06:18.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:06:18.201 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:06:18.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:18.202 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:18.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:06:18.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:06:18.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:06:18.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-07-26T07:06:18.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:06:18.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:18.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:18.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257957 2021-07-26T07:06:18.209 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:06:18.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-07-26T07:06:18.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-07-26T07:06:18.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:06:18.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-07-26T07:06:18.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2021-07-26T07:06:18.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2021-07-26T07:06:18.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2021-07-26T07:06:18.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2021-07-26T07:06:18.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2021-07-26T07:06:18.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2021-07-26T07:06:18.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2021-07-26T07:06:18.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-07-26T07:06:18.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:06:18.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:06:18.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:06:18.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-07-26T07:06:18.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-07-26T07:06:18.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:06:18.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:18.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:18.597 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:18.597 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:18.598 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:18.598 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:18.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:06:18.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:06:18.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:06:18.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:06:18.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:06:18.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:06:18.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:06:18.919 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:06:18.920 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:06:18.920 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:06:18.921 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:06:18.921 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:18.921 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:18.922 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:06:18.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:06:18.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get fsid 2021-07-26T07:06:19.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:06:19.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:06:19.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:06:19.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:06:19.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:06:19.012 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:06:19.012 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:06:19.012 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:06:19.013 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:06:19.014 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:19.014 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:19.014 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:06:19.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:06:19.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get mon_host 2021-07-26T07:06:19.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2021-07-26T07:06:19.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2021-07-26T07:06:19.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:06:19.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:06:19.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:06:19.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2021-07-26T07:06:19.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:06:19.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:06:19.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:19.979 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:19.979 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:19.980 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:19.980 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:19.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:19.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:06:19.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:06:20.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2021-07-26T07:06:20.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:06:20.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:06:20.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:06:20.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:06:20.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:06:20.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:06:20.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:20.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:20.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:20.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:06:20.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:06:20.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:20.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:20.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:06:20.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:06:20.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:06:20.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:20.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:20.021 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:20.022 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:20.022 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:20.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:20.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:20.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:20.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:06:20.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:06:20.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:06:20.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:06:20.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:20.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:20.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:20.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:06:20.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:06:20.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:06:20.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:06:20.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9cdfd378-1f90-4da8-a122-4bce8d87fa77 2021-07-26T07:06:20.026 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 9cdfd378-1f90-4da8-a122-4bce8d87fa77 2021-07-26T07:06:20.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9cdfd378-1f90-4da8-a122-4bce8d87fa77' 2021-07-26T07:06:20.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:06:20.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsXv5g9Dq0ARAAvXdOo0UliruU0jeC3jRgsQ== 2021-07-26T07:06:20.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsXv5g9Dq0ARAAvXdOo0UliruU0jeC3jRgsQ=="}' 2021-07-26T07:06:20.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9cdfd378-1f90-4da8-a122-4bce8d87fa77 -i td/osd-recovery-stats/0/new.json 2021-07-26T07:06:20.394 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:06:20.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2021-07-26T07:06:20.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDsXv5g9Dq0ARAAvXdOo0UliruU0jeC3jRgsQ== --osd-uuid 9cdfd378-1f90-4da8-a122-4bce8d87fa77 2021-07-26T07:06:20.437 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:20.426+0000 7fbabab15f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:20.440 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:20.430+0000 7fbabab15f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:20.442 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:20.430+0000 7fbabab15f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:20.442 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:20.430+0000 7fbabab15f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:06:20.443 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:20.430+0000 7fbabab15f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:06:20.443 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:20.430+0000 7fbabab15f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:06:20.521 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:20.510+0000 7fbabab15f40 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-07-26T07:06:26.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-07-26T07:06:26.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:06:26.183 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:06:26.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:06:26.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:06:26.667 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:06:26.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:06:26.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:06:26.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:06:26.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:06:26.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:26.695 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:26.681+0000 7f0337dc0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:26.697 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:26.685+0000 7f0337dc0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:26.700 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:26.685+0000 7f0337dc0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:26.984 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:26.969+0000 7f0337dc0f40 -1 Falling back to public interface 2021-07-26T07:06:27.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:06:27.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:27.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:06:27.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:27.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:27.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:27.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:27.009 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:06:27.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:27.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:27.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:28.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:28.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:28.396 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:06:28.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:28.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:28.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:28.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:29.543 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:29.529+0000 7f0337dc0f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:06:29.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:29.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:29.758 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:06:29.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:29.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:29.760 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:30.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:31.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:31.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:31.194 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:06:31.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:31.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:31.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:06:31.556 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/259061,v1:127.0.0.1:6803/259061] [v2:127.0.0.1:6804/259061,v1:127.0.0.1:6805/259061] exists,up 9cdfd378-1f90-4da8-a122-4bce8d87fa77 2021-07-26T07:06:31.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:31.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:31.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:31.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2021-07-26T07:06:31.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:06:31.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:06:31.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:06:31.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:06:31.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:06:31.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:06:31.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:31.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:31.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:31.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:06:31.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:06:31.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:31.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:31.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:06:31.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:06:31.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:06:31.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:31.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:31.567 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:31.567 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:31.568 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:31.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:31.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:31.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:06:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:06:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:06:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:06:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:31.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:31.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:31.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:06:31.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:06:31.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:06:31.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:06:31.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=873c64e3-5a58-4a3f-b744-2ad27165f448 2021-07-26T07:06:31.572 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 873c64e3-5a58-4a3f-b744-2ad27165f448 2021-07-26T07:06:31.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 873c64e3-5a58-4a3f-b744-2ad27165f448' 2021-07-26T07:06:31.573 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:06:31.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3Xv5gzD0lIhAAPr/Uvco3o0j57bVJSKDYYg== 2021-07-26T07:06:31.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3Xv5gzD0lIhAAPr/Uvco3o0j57bVJSKDYYg=="}' 2021-07-26T07:06:31.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 873c64e3-5a58-4a3f-b744-2ad27165f448 -i td/osd-recovery-stats/1/new.json 2021-07-26T07:06:32.135 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:06:32.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2021-07-26T07:06:32.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD3Xv5gzD0lIhAAPr/Uvco3o0j57bVJSKDYYg== --osd-uuid 873c64e3-5a58-4a3f-b744-2ad27165f448 2021-07-26T07:06:32.181 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:32.169+0000 7fefeb3fcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:32.204 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:32.193+0000 7fefeb3fcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:32.207 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:32.193+0000 7fefeb3fcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:32.207 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:32.193+0000 7fefeb3fcf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:06:32.208 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:32.193+0000 7fefeb3fcf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:06:32.208 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:32.193+0000 7fefeb3fcf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:06:32.308 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:32.293+0000 7fefeb3fcf40 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-07-26T07:06:38.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-07-26T07:06:38.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:06:38.155 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:06:38.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:06:38.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:06:38.662 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:06:38.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:06:38.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:38.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:06:38.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:06:38.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:06:38.692 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:38.681+0000 7f74b4161f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:38.695 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:38.681+0000 7f74b4161f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:38.696 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:38.685+0000 7f74b4161f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:39.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:06:39.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:39.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:06:39.020 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:06:39.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:39.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:39.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:39.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:39.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:39.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:39.255 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:39.241+0000 7f74b4161f40 -1 Falling back to public interface 2021-07-26T07:06:39.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:40.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:40.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:40.390 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:06:40.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:40.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:40.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:40.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:41.398 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:41.385+0000 7f74b4161f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:06:41.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:41.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:41.743 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:06:41.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:41.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:41.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:42.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:43.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:43.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:43.122 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:06:43.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:43.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:43.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:06:43.493 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/260467,v1:127.0.0.1:6811/260467] [v2:127.0.0.1:6812/260467,v1:127.0.0.1:6813/260467] exists,up 873c64e3-5a58-4a3f-b744-2ad27165f448 2021-07-26T07:06:43.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:43.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:43.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2021-07-26T07:06:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:06:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:06:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:06:43.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:06:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2021-07-26T07:06:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:06:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:43.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:43.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:43.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-07-26T07:06:43.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-07-26T07:06:43.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:43.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:43.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:06:43.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:06:43.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:06:43.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:43.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:43.500 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:43.500 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:43.500 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:43.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:43.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:43.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:43.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:06:43.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:06:43.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:06:43.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:06:43.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:06:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:06:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2021-07-26T07:06:43.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:06:43.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89135fcf-2f12-4591-9b14-de73809f7a8f 2021-07-26T07:06:43.505 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 89135fcf-2f12-4591-9b14-de73809f7a8f 2021-07-26T07:06:43.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 89135fcf-2f12-4591-9b14-de73809f7a8f' 2021-07-26T07:06:43.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:06:43.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADX/5gEsEyHhAAJ3WVLjVF9G7t+6T5mrSLmg== 2021-07-26T07:06:43.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADX/5gEsEyHhAAJ3WVLjVF9G7t+6T5mrSLmg=="}' 2021-07-26T07:06:43.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89135fcf-2f12-4591-9b14-de73809f7a8f -i td/osd-recovery-stats/2/new.json 2021-07-26T07:06:44.025 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:06:44.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2021-07-26T07:06:44.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQADX/5gEsEyHhAAJ3WVLjVF9G7t+6T5mrSLmg== --osd-uuid 89135fcf-2f12-4591-9b14-de73809f7a8f 2021-07-26T07:06:44.074 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:44.061+0000 7f84fd538f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:44.100 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:44.089+0000 7f84fd538f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:44.102 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:44.089+0000 7f84fd538f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:44.103 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:44.089+0000 7f84fd538f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:06:44.104 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:44.089+0000 7f84fd538f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:06:44.104 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:44.089+0000 7f84fd538f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:06:44.204 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:44.193+0000 7f84fd538f40 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-07-26T07:06:50.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-07-26T07:06:50.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:06:50.072 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T07:06:50.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:06:50.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:06:50.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:06:50.596 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T07:06:50.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:50.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:06:50.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:06:50.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:06:50.627 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:50.613+0000 7f2287561f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:50.630 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:50.617+0000 7f2287561f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:50.630 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:50.617+0000 7f2287561f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:50.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:06:50.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:06:50.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:06:50.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:06:50.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:06:50.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:50.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:06:50.942 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:06:50.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:50.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:06:51.178 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:51.165+0000 7f2287561f40 -1 Falling back to public interface 2021-07-26T07:06:51.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:52.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:52.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:52.324 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:06:52.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:06:52.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:52.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:06:52.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:53.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:53.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:53.686 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:06:53.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:06:53.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:53.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:06:53.848 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:53.837+0000 7f2287561f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:06:54.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:06:55.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:06:55.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:06:55.114 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:06:55.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:06:55.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:06:55.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:06:55.505 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/261873,v1:127.0.0.1:6819/261873] [v2:127.0.0.1:6820/261873,v1:127.0.0.1:6821/261873] exists,up 89135fcf-2f12-4591-9b14-de73809f7a8f 2021-07-26T07:06:55.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:06:55.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:06:55.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:06:55.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2021-07-26T07:06:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:06:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:06:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T07:06:55.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:06:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2021-07-26T07:06:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:06:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:06:55.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:06:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:06:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-07-26T07:06:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-07-26T07:06:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:06:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:06:55.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:06:55.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:06:55.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:06:55.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:06:55.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:06:55.514 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:06:55.514 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:06:55.515 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:06:55.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:55.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:06:55.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:06:55.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:06:55.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:06:55.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:06:55.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:06:55.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:06:55.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:06:55.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:06:55.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:06:55.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:06:55.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2021-07-26T07:06:55.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:06:55.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b005db76-cee6-4197-b8bd-9a50f617a5ba 2021-07-26T07:06:55.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b005db76-cee6-4197-b8bd-9a50f617a5ba' 2021-07-26T07:06:55.522 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 b005db76-cee6-4197-b8bd-9a50f617a5ba 2021-07-26T07:06:55.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:06:55.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPX/5gSKRGHxAAFAMIDgxrFfMEfv41ueQNmQ== 2021-07-26T07:06:55.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPX/5gSKRGHxAAFAMIDgxrFfMEfv41ueQNmQ=="}' 2021-07-26T07:06:55.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b005db76-cee6-4197-b8bd-9a50f617a5ba -i td/osd-recovery-stats/3/new.json 2021-07-26T07:06:56.135 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:06:56.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2021-07-26T07:06:56.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAPX/5gSKRGHxAAFAMIDgxrFfMEfv41ueQNmQ== --osd-uuid b005db76-cee6-4197-b8bd-9a50f617a5ba 2021-07-26T07:06:56.187 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:56.173+0000 7f27e568cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:56.217 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:56.205+0000 7f27e568cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:56.219 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:56.205+0000 7f27e568cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:06:56.220 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:56.209+0000 7f27e568cf40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:06:56.221 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:56.209+0000 7f27e568cf40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:06:56.221 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:56.209+0000 7f27e568cf40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:06:56.313 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:06:56.301+0000 7f27e568cf40 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-07-26T07:07:02.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-07-26T07:07:02.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:07:02.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T07:07:02.172 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T07:07:02.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:07:02.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T07:07:02.676 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T07:07:02.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:02.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:07:02.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:07:02.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:07:02.703 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:02.689+0000 7ff50211ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:02.705 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:02.693+0000 7ff50211ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:02.709 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:02.693+0000 7ff50211ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:03.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T07:07:03.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:03.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:07:03.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:03.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:03.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:03.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:03.006 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:07:03.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:07:03.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:03.254 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:03.241+0000 7ff50211ef40 -1 Falling back to public interface 2021-07-26T07:07:03.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:04.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:04.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:04.339 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:07:04.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:04.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:04.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:07:04.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:05.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:05.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:05.706 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:07:05.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:05.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:05.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:07:05.812 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:05.797+0000 7ff50211ef40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:07:06.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:07.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:07.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:07.062 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:07:07.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:07.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:07.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:07:07.470 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/263279,v1:127.0.0.1:6827/263279] [v2:127.0.0.1:6828/263279,v1:127.0.0.1:6829/263279] exists,up b005db76-cee6-4197-b8bd-9a50f617a5ba 2021-07-26T07:07:07.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:07.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:07.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:07.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2021-07-26T07:07:07.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:07:07.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:07:07.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T07:07:07.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:07:07.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2021-07-26T07:07:07.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:07:07.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:07:07.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:07:07.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:07:07.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-07-26T07:07:07.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-07-26T07:07:07.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:07:07.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:07:07.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:07:07.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:07:07.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:07:07.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:07:07.488 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:07:07.489 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:07:07.489 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:07.490 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:07:07.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:07:07.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:07:07.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:07:07.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:07:07.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:07:07.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:07:07.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:07:07.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:07:07.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:07:07.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:07.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:07:07.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:07:07.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2021-07-26T07:07:07.495 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:07:07.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f9f749c5-8c68-4d0d-9ce5-c5b21593a17e 2021-07-26T07:07:07.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 f9f749c5-8c68-4d0d-9ce5-c5b21593a17e' 2021-07-26T07:07:07.497 INFO:tasks.workunit.client.0.smithi094.stdout:add osd4 f9f749c5-8c68-4d0d-9ce5-c5b21593a17e 2021-07-26T07:07:07.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:07:07.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbX/5g4vkyHRAACjSGvVIfjn1V1N3IOEpEoQ== 2021-07-26T07:07:07.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbX/5g4vkyHRAACjSGvVIfjn1V1N3IOEpEoQ=="}' 2021-07-26T07:07:07.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f9f749c5-8c68-4d0d-9ce5-c5b21593a17e -i td/osd-recovery-stats/4/new.json 2021-07-26T07:07:08.101 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T07:07:08.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2021-07-26T07:07:08.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAbX/5g4vkyHRAACjSGvVIfjn1V1N3IOEpEoQ== --osd-uuid f9f749c5-8c68-4d0d-9ce5-c5b21593a17e 2021-07-26T07:07:08.156 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:08.141+0000 7f95f031df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:08.185 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:08.173+0000 7f95f031df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:08.188 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:08.173+0000 7f95f031df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:08.188 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:08.177+0000 7f95f031df40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:07:08.189 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:08.177+0000 7f95f031df40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:07:08.189 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:08.177+0000 7f95f031df40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:07:08.276 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:08.261+0000 7f95f031df40 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-07-26T07:07:14.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-07-26T07:07:14.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:07:14.727 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd4 key to auth repository 2021-07-26T07:07:14.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T07:07:14.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:07:15.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T07:07:15.233 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.4 2021-07-26T07:07:15.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:07:15.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:15.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:07:15.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:07:15.262 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:15.248+0000 7f392871bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:15.264 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:15.252+0000 7f392871bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:15.272 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:15.252+0000 7f392871bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:15.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T07:07:15.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:15.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T07:07:15.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:15.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:15.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:15.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:15.570 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:07:15.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:15.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:07:15.814 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:15.800+0000 7f392871bf40 -1 Falling back to public interface 2021-07-26T07:07:15.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:16.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:16.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:16.964 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:07:16.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:16.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:16.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:07:17.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:18.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:18.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:18.331 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:07:18.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:18.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:18.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:07:18.484 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:18.468+0000 7f392871bf40 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T07:07:18.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:19.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:19.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:19.737 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:07:19.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:19.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:19.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:07:20.135 INFO:tasks.workunit.client.0.smithi094.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/264686,v1:127.0.0.1:6835/264686] [v2:127.0.0.1:6836/264686,v1:127.0.0.1:6837/264686] exists,up f9f749c5-8c68-4d0d-9ce5-c5b21593a17e 2021-07-26T07:07:20.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:20.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:20.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:20.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2021-07-26T07:07:20.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:07:20.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:07:20.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T07:07:20.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:07:20.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2021-07-26T07:07:20.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:07:20.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:07:20.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:07:20.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:07:20.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-07-26T07:07:20.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-07-26T07:07:20.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:07:20.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:07:20.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:07:20.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:07:20.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:07:20.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:07:20.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:07:20.148 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:07:20.148 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:07:20.149 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:07:20.149 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:07:20.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:07:20.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:07:20.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:07:20.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:07:20.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:07:20.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:07:20.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:07:20.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:07:20.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:20.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:07:20.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:07:20.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2021-07-26T07:07:20.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:07:20.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7cff27d2-b888-4795-816b-2969dd2cadd8 2021-07-26T07:07:20.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 7cff27d2-b888-4795-816b-2969dd2cadd8' 2021-07-26T07:07:20.153 INFO:tasks.workunit.client.0.smithi094.stdout:add osd5 7cff27d2-b888-4795-816b-2969dd2cadd8 2021-07-26T07:07:20.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:07:20.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAoX/5gy+A3CRAAztoQPEgD1UdIeiYUQ0rxYQ== 2021-07-26T07:07:20.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAoX/5gy+A3CRAAztoQPEgD1UdIeiYUQ0rxYQ=="}' 2021-07-26T07:07:20.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7cff27d2-b888-4795-816b-2969dd2cadd8 -i td/osd-recovery-stats/5/new.json 2021-07-26T07:07:20.702 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T07:07:20.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2021-07-26T07:07:20.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAoX/5gy+A3CRAAztoQPEgD1UdIeiYUQ0rxYQ== --osd-uuid 7cff27d2-b888-4795-816b-2969dd2cadd8 2021-07-26T07:07:20.752 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:20.736+0000 7fd2b98b7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:20.783 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:20.768+0000 7fd2b98b7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:20.785 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:20.772+0000 7fd2b98b7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:20.786 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:20.772+0000 7fd2b98b7f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:07:20.786 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:20.772+0000 7fd2b98b7f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:07:20.787 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:20.772+0000 7fd2b98b7f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:07:20.952 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:20.940+0000 7fd2b98b7f40 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-07-26T07:07:27.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-07-26T07:07:27.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:07:27.116 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd5 key to auth repository 2021-07-26T07:07:27.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T07:07:27.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:07:27.605 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.5 2021-07-26T07:07:27.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T07:07:27.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:07:27.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:07:27.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:07:27.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:07:27.633 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:27.620+0000 7fb1d96cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:27.635 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:27.620+0000 7fb1d96cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:27.637 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:27.624+0000 7fb1d96cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:07:27.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T07:07:27.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:07:27.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T07:07:27.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:07:27.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:07:27.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:27.961 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:07:27.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:07:27.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:27.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:07:28.175 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:28.160+0000 7fb1d96cff40 -1 Falling back to public interface 2021-07-26T07:07:28.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:29.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:29.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:29.313 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:07:29.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:07:29.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:29.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:07:29.660 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:30.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:30.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:30.663 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:07:30.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:07:30.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:30.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:07:30.710 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:07:30.696+0000 7fb1d96cff40 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T07:07:31.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:07:32.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:07:32.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:07:32.033 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:07:32.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:07:32.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:07:32.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:07:32.393 INFO:tasks.workunit.client.0.smithi094.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/266092,v1:127.0.0.1:6843/266092] [v2:127.0.0.1:6844/266092,v1:127.0.0.1:6845/266092] exists,up 7cff27d2-b888-4795-816b-2969dd2cadd8 2021-07-26T07:07:32.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:07:32.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:07:32.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:07:32.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2021-07-26T07:07:32.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-07-26T07:07:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2021-07-26T07:07:33.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-07-26T07:07:34.068 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T07:07:34.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:07:35.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2021-07-26T07:07:35.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:07:35.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:07:35.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:07:35.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:07:35.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:07:35.081 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:07:35.081 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:07:35.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:07:35.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:07:35.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:07:35.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:07:35.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:07:35.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:07:35.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:07:35.289 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:07:35.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:07:35.647 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:07:35.647 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:07:35.647 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:07:35.648 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:07:35.648 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:07:35.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:07:35.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:35.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:07:35.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836496 2021-07-26T07:07:35.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836496 2021-07-26T07:07:35.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496' 2021-07-26T07:07:35.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:35.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:07:35.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672973 2021-07-26T07:07:35.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672973 2021-07-26T07:07:35.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973' 2021-07-26T07:07:35.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:35.964 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:07:36.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509451 2021-07-26T07:07:36.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509451 2021-07-26T07:07:36.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451' 2021-07-26T07:07:36.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:36.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:07:36.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345928 2021-07-26T07:07:36.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345928 2021-07-26T07:07:36.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451 3-85899345928' 2021-07-26T07:07:36.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:36.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:07:36.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182406 2021-07-26T07:07:36.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182406 2021-07-26T07:07:36.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451 3-85899345928 4-107374182406' 2021-07-26T07:07:36.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:07:36.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:07:36.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018883 2021-07-26T07:07:36.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018883 2021-07-26T07:07:36.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451 3-85899345928 4-107374182406 5-128849018883' 2021-07-26T07:07:36.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:36.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836496 2021-07-26T07:07:36.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:36.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:07:36.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836496 2021-07-26T07:07:36.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:36.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836496 2021-07-26T07:07:36.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-07-26T07:07:36.540 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836496 2021-07-26T07:07:36.541 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:07:36.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836497 -lt 21474836496 2021-07-26T07:07:36.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:36.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672973 2021-07-26T07:07:36.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:36.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:07:36.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672973 2021-07-26T07:07:36.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:36.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672973 2021-07-26T07:07:36.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-07-26T07:07:36.906 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672973 2021-07-26T07:07:36.906 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:07:37.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672973 -lt 42949672973 2021-07-26T07:07:37.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:37.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509451 2021-07-26T07:07:37.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:37.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:07:37.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509451 2021-07-26T07:07:37.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:37.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509451 2021-07-26T07:07:37.261 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509451 2021-07-26T07:07:37.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-07-26T07:07:37.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:07:37.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509449 -lt 64424509451 2021-07-26T07:07:37.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:07:38.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:07:38.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:07:38.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509451 -lt 64424509451 2021-07-26T07:07:38.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:38.969 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345928 2021-07-26T07:07:38.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:38.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:07:38.974 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345928 2021-07-26T07:07:38.974 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:38.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345928 2021-07-26T07:07:38.976 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345928 2021-07-26T07:07:38.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2021-07-26T07:07:38.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:07:39.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345929 -lt 85899345928 2021-07-26T07:07:39.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:39.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182406 2021-07-26T07:07:39.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:39.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:07:39.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182406 2021-07-26T07:07:39.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:39.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182406 2021-07-26T07:07:39.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-07-26T07:07:39.392 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182406 2021-07-26T07:07:39.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:07:39.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182406 2021-07-26T07:07:39.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:07:39.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018883 2021-07-26T07:07:39.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:07:39.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:07:39.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018883 2021-07-26T07:07:39.768 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:07:39.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018883 2021-07-26T07:07:39.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-07-26T07:07:39.770 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018883 2021-07-26T07:07:39.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:07:40.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018884 -lt 128849018883 2021-07-26T07:07:40.189 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:07:40.189 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:40.189 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:40.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:07:40.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:07:40.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:07:40.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:07:40.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:07:40.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:07:40.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:07:40.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:07:41.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:07:41.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:07:41.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:07:41.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:07:41.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:07:41.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:07:41.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:07:41.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2021-07-26T07:07:41.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:41.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2021-07-26T07:07:41.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:41.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2021-07-26T07:07:41.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:41.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2021-07-26T07:07:42.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:42.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2021-07-26T07:07:42.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:42.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2021-07-26T07:07:42.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:42.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2021-07-26T07:07:42.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:42.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2021-07-26T07:07:42.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:42.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2021-07-26T07:07:43.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:43.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2021-07-26T07:07:43.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:43.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2021-07-26T07:07:43.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:43.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2021-07-26T07:07:43.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:43.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2021-07-26T07:07:43.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:43.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2021-07-26T07:07:44.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:44.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2021-07-26T07:07:44.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:44.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2021-07-26T07:07:44.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:44.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2021-07-26T07:07:44.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:44.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2021-07-26T07:07:44.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:44.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2021-07-26T07:07:45.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:45.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2021-07-26T07:07:45.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:45.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2021-07-26T07:07:45.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:45.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2021-07-26T07:07:45.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:45.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2021-07-26T07:07:45.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:45.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2021-07-26T07:07:46.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:46.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2021-07-26T07:07:46.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:46.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2021-07-26T07:07:46.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:46.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2021-07-26T07:07:46.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:46.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2021-07-26T07:07:46.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:46.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2021-07-26T07:07:47.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:47.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2021-07-26T07:07:47.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:47.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2021-07-26T07:07:47.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:47.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2021-07-26T07:07:47.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:47.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2021-07-26T07:07:47.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:47.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2021-07-26T07:07:47.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:47.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2021-07-26T07:07:48.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:48.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2021-07-26T07:07:48.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:48.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2021-07-26T07:07:48.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:48.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2021-07-26T07:07:48.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:48.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2021-07-26T07:07:48.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:48.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2021-07-26T07:07:49.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:49.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2021-07-26T07:07:49.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:49.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2021-07-26T07:07:49.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:49.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2021-07-26T07:07:49.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:49.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2021-07-26T07:07:50.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:50.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2021-07-26T07:07:50.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:50.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2021-07-26T07:07:50.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:50.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2021-07-26T07:07:50.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:50.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2021-07-26T07:07:50.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:50.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2021-07-26T07:07:51.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:51.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2021-07-26T07:07:51.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:51.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2021-07-26T07:07:51.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:51.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2021-07-26T07:07:51.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:51.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2021-07-26T07:07:51.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:51.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2021-07-26T07:07:51.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:51.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2021-07-26T07:07:52.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:52.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2021-07-26T07:07:52.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:52.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2021-07-26T07:07:52.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:52.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2021-07-26T07:07:52.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:52.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2021-07-26T07:07:52.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:52.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2021-07-26T07:07:53.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:53.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2021-07-26T07:07:53.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:53.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2021-07-26T07:07:53.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:53.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2021-07-26T07:07:53.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:53.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2021-07-26T07:07:53.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:53.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2021-07-26T07:07:53.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:53.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2021-07-26T07:07:54.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:54.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2021-07-26T07:07:54.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:54.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2021-07-26T07:07:54.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:54.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2021-07-26T07:07:55.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:55.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2021-07-26T07:07:55.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:55.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2021-07-26T07:07:55.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:55.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2021-07-26T07:07:55.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:55.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2021-07-26T07:07:55.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:55.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2021-07-26T07:07:56.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:56.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2021-07-26T07:07:56.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:56.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2021-07-26T07:07:56.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:56.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2021-07-26T07:07:56.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:56.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2021-07-26T07:07:56.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:56.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2021-07-26T07:07:56.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:56.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2021-07-26T07:07:57.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:57.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2021-07-26T07:07:57.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:57.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2021-07-26T07:07:57.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:57.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2021-07-26T07:07:57.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:57.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2021-07-26T07:07:57.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:57.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2021-07-26T07:07:58.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:58.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2021-07-26T07:07:58.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:58.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2021-07-26T07:07:58.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:58.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2021-07-26T07:07:58.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:58.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2021-07-26T07:07:59.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:59.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2021-07-26T07:07:59.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:59.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2021-07-26T07:07:59.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:59.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2021-07-26T07:07:59.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:59.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2021-07-26T07:07:59.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:59.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2021-07-26T07:07:59.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:07:59.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2021-07-26T07:08:00.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:00.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2021-07-26T07:08:00.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:00.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2021-07-26T07:08:00.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:00.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2021-07-26T07:08:00.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:00.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2021-07-26T07:08:01.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:01.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2021-07-26T07:08:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:01.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2021-07-26T07:08:01.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:01.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2021-07-26T07:08:01.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:01.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2021-07-26T07:08:01.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:01.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2021-07-26T07:08:01.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:01.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2021-07-26T07:08:02.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:02.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2021-07-26T07:08:02.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:02.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2021-07-26T07:08:02.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:02.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2021-07-26T07:08:02.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:02.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2021-07-26T07:08:03.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:03.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2021-07-26T07:08:03.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:03.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2021-07-26T07:08:03.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:03.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2021-07-26T07:08:03.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:03.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2021-07-26T07:08:03.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:03.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2021-07-26T07:08:03.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:03.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2021-07-26T07:08:04.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:04.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2021-07-26T07:08:04.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:04.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2021-07-26T07:08:04.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:04.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2021-07-26T07:08:04.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:04.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2021-07-26T07:08:04.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:04.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2021-07-26T07:08:05.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:05.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2021-07-26T07:08:05.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:05.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2021-07-26T07:08:05.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:05.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2021-07-26T07:08:05.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:05.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2021-07-26T07:08:05.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:05.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2021-07-26T07:08:06.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:06.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2021-07-26T07:08:06.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:06.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2021-07-26T07:08:06.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:06.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2021-07-26T07:08:06.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:06.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2021-07-26T07:08:06.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:06.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2021-07-26T07:08:07.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:07.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2021-07-26T07:08:07.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:07.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2021-07-26T07:08:07.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:07.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2021-07-26T07:08:07.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:07.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2021-07-26T07:08:07.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:07.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2021-07-26T07:08:08.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:08.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2021-07-26T07:08:08.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2021-07-26T07:08:08.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:08.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2021-07-26T07:08:08.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:08.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2021-07-26T07:08:09.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:09.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2021-07-26T07:08:09.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:09.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2021-07-26T07:08:09.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:09.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2021-07-26T07:08:09.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:09.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2021-07-26T07:08:09.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:09.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2021-07-26T07:08:10.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:10.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2021-07-26T07:08:10.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:10.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2021-07-26T07:08:10.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:10.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2021-07-26T07:08:10.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:10.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2021-07-26T07:08:10.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:10.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2021-07-26T07:08:11.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:11.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2021-07-26T07:08:11.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:11.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2021-07-26T07:08:11.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:11.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2021-07-26T07:08:11.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:11.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2021-07-26T07:08:11.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:11.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2021-07-26T07:08:12.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:12.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2021-07-26T07:08:12.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:12.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2021-07-26T07:08:12.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:12.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2021-07-26T07:08:12.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:12.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2021-07-26T07:08:12.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:12.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2021-07-26T07:08:13.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:13.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2021-07-26T07:08:13.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:13.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2021-07-26T07:08:13.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:13.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2021-07-26T07:08:13.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:13.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2021-07-26T07:08:13.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:13.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2021-07-26T07:08:14.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:14.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2021-07-26T07:08:14.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:14.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2021-07-26T07:08:14.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:14.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2021-07-26T07:08:14.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:14.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2021-07-26T07:08:14.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:14.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2021-07-26T07:08:15.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:15.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2021-07-26T07:08:15.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:15.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2021-07-26T07:08:15.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:15.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2021-07-26T07:08:15.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:15.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2021-07-26T07:08:15.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:15.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2021-07-26T07:08:16.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:16.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2021-07-26T07:08:16.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:16.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2021-07-26T07:08:16.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:16.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2021-07-26T07:08:16.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:16.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2021-07-26T07:08:16.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:16.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2021-07-26T07:08:17.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:17.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2021-07-26T07:08:17.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:17.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2021-07-26T07:08:17.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:17.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2021-07-26T07:08:17.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:17.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2021-07-26T07:08:17.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:17.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2021-07-26T07:08:18.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:18.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2021-07-26T07:08:18.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:18.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2021-07-26T07:08:18.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:18.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2021-07-26T07:08:18.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:18.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2021-07-26T07:08:19.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:19.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2021-07-26T07:08:19.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:19.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2021-07-26T07:08:19.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:19.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2021-07-26T07:08:19.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:19.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2021-07-26T07:08:19.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:19.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2021-07-26T07:08:20.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:20.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2021-07-26T07:08:20.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:20.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2021-07-26T07:08:20.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:20.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2021-07-26T07:08:20.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:20.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2021-07-26T07:08:20.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:20.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2021-07-26T07:08:21.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:21.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2021-07-26T07:08:21.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:21.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2021-07-26T07:08:21.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:08:21.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2021-07-26T07:08:21.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2021-07-26T07:08:21.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:08:21.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:08:21.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:08:21.780 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:08:22.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2021-07-26T07:08:22.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2021-07-26T07:08:22.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T07:08:22.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:08:22.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test obj1 2021-07-26T07:08:22.148 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:08:22.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2021-07-26T07:08:22.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2021-07-26T07:08:22.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test 2021-07-26T07:08:22.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T07:08:22.535 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test obj1 2021-07-26T07:08:22.535 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:08:22.535 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:08:22.535 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:08:22.536 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:08:22.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:08:22.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test obj1 2021-07-26T07:08:22.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:08:23.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2021-07-26T07:08:23.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2021-07-26T07:08:23.972 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is set 2021-07-26T07:08:23.985 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2021-07-26T07:08:23.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 259061 2021-07-26T07:08:23.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2021-07-26T07:08:24.417 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T07:08:24.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2021-07-26T07:08:25.298 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T07:08:25.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2021-07-26T07:08:26.493 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is unset 2021-07-26T07:08:26.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2021-07-26T07:08:26.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:08:26.507 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:08:26.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:08:26.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:08:26.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2021-07-26T07:08:27.012 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T07:08:27.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2021-07-26T07:08:29.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2021-07-26T07:08:29.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:08:29.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:08:29.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:08:29.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:08:29.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:08:29.025 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:08:29.025 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:08:29.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:08:29.026 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:08:29.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:08:29.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:08:29.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:08:29.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:08:29.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:08:29.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:08:29.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:08:29.612 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:08:29.613 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:08:29.613 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:08:29.613 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:08:29.613 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:08:29.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:08:29.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:29.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:08:29.715 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-07-26T07:08:29.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T07:08:29.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T07:08:29.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T07:08:29.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:29.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:08:29.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672986 2021-07-26T07:08:29.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672986 2021-07-26T07:08:29.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672986' 2021-07-26T07:08:29.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:29.886 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:08:30.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509463 2021-07-26T07:08:30.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509463 2021-07-26T07:08:30.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672986 2-64424509463' 2021-07-26T07:08:30.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:30.028 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:08:30.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345941 2021-07-26T07:08:30.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345941 2021-07-26T07:08:30.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672986 2-64424509463 3-85899345941' 2021-07-26T07:08:30.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:30.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:08:30.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182419 2021-07-26T07:08:30.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182419 2021-07-26T07:08:30.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672986 2-64424509463 3-85899345941 4-107374182419' 2021-07-26T07:08:30.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:08:30.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:08:30.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018896 2021-07-26T07:08:30.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018896 2021-07-26T07:08:30.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672986 2-64424509463 3-85899345941 4-107374182419 5-128849018896' 2021-07-26T07:08:30.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:30.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672986 2021-07-26T07:08:30.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:30.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:08:30.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672986 2021-07-26T07:08:30.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:30.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672986 2021-07-26T07:08:30.473 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672986 2021-07-26T07:08:30.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2021-07-26T07:08:30.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:08:30.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672986 -lt 42949672986 2021-07-26T07:08:30.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:30.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509463 2021-07-26T07:08:30.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:30.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:08:30.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509463 2021-07-26T07:08:30.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:30.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509463 2021-07-26T07:08:30.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2021-07-26T07:08:30.845 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509463 2021-07-26T07:08:30.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:08:31.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509461 -lt 64424509463 2021-07-26T07:08:31.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:08:32.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:08:32.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:08:32.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509464 -lt 64424509463 2021-07-26T07:08:32.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:32.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345941 2021-07-26T07:08:32.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:32.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:08:32.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345941 2021-07-26T07:08:32.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:32.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345941 2021-07-26T07:08:32.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2021-07-26T07:08:32.606 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345941 2021-07-26T07:08:32.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:08:33.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345941 -lt 85899345941 2021-07-26T07:08:33.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:33.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182419 2021-07-26T07:08:33.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:33.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:08:33.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182419 2021-07-26T07:08:33.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:33.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182419 2021-07-26T07:08:33.014 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182419 2021-07-26T07:08:33.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2021-07-26T07:08:33.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:08:33.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182419 -lt 107374182419 2021-07-26T07:08:33.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:08:33.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018896 2021-07-26T07:08:33.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:08:33.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:08:33.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018896 2021-07-26T07:08:33.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:08:33.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018896 2021-07-26T07:08:33.483 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018896 2021-07-26T07:08:33.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018896' 2021-07-26T07:08:33.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:08:33.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018896 -lt 128849018896 2021-07-26T07:08:33.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:08:33.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:33.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:34.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:08:34.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:34.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:34.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:34.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:34.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:34.322 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:34.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:34.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:34.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:34.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:34.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:35.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:35.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:08:35.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:08:35.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:08:35.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:35.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:35.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:35.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:35.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:35.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:35.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:35.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:35.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:35.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:35.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:35.703 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:35.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:35.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:36.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:36.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:36.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:36.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:36.213 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:36.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:08:36.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:08:36.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:36.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:36.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:36.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:36.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:36.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:36.766 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:36.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:36.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:36.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:36.768 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:37.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:37.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:37.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:37.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:37.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:37.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:37.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:37.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:37.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:37.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:37.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:37.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:37.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:38.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:38.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:38.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:38.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:38.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:38.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:38.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:38.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:38.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:38.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:38.261 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:38.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:38.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:38.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:38.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:38.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:38.667 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:39.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:39.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:39.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:39.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:39.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:39.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:39.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:39.150 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:39.150 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:39.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:39.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:39.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:39.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:39.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:39.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:39.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:39.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:39.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:39.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:39.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:39.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:39.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:40.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:40.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:40.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:40.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:40.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:40.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:40.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:40.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:40.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:40.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:40.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:40.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:40.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:40.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:40.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:40.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:40.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:40.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:41.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:41.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:41.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:41.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:41.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:41.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:41.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:41.045 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:41.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:41.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:41.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:41.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:41.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:41.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:41.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:41.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:41.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:41.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:41.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:41.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:41.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:42.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:08:42.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:08:42.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:42.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:42.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:42.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:42.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:42.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:42.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:42.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:42.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:42.408 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:42.408 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:42.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:42.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:42.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:42.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:43.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:43.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:43.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:43.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:43.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:43.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:43.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:43.234 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:43.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:43.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:08:43.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:08:43.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:43.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:43.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:43.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:43.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:43.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:43.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:43.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:43.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:43.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:43.900 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:44.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:44.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:44.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:44.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:44.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:44.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:44.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:44.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:44.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:44.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:44.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:44.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:44.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:45.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=6 2021-07-26T07:08:45.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 6 '!=' null 2021-07-26T07:08:45.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:45.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:45.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:45.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:45.322 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:45.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:45.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:45.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:45.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:45.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:45.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:45.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:45.681 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:45.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:45.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:46.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:46.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:46.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:46.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:46.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:46.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:46.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:46.139 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:46.139 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:46.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:46.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:46.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:46.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:46.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:46.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:46.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:46.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:46.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:46.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:46.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:46.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:46.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:47.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:47.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:47.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:47.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:47.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:47.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:47.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:47.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:47.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:47.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:47.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:47.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:47.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:48.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:08:48.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:08:48.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:48.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:48.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:48.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:48.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:48.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:48.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:48.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:48.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:48.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:48.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:48.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:48.629 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:48.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:48.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:49.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:49.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:49.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:49.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:49.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:49.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:49.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:49.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:49.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:49.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:08:49.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:08:49.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:49.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:49.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:49.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:49.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:49.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:49.652 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:49.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:49.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:49.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:49.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:50.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:50.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:50.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:50.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:50.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:50.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:50.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:50.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:50.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:50.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:50.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:50.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:50.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:50.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:50.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:50.963 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:50.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:50.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:51.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:51.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:51.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:51.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:51.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:51.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:51.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:51.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:51.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:51.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:51.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:51.440 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:51.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:51.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:51.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:51.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:51.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:51.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:51.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:51.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:51.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:52.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:52.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:52.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:52.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:52.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:52.506 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:52.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:52.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:52.508 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:52.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:52.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:52.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:52.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:52.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:52.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:52.862 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:52.863 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:53.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:53.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:53.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:53.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:53.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:53.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:53.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:53.430 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:53.431 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:53.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:53.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:53.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:53.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:53.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:54.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:54.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:54.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:54.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:54.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:54.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:54.093 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:54.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:54.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:54.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:54.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:54.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:54.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:54.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:54.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:54.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:54.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:54.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:54.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:54.954 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:54.954 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:55.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:08:55.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:08:55.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:55.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:55.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:55.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:55.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:55.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:55.614 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:55.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:55.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:55.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:55.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:55.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:55.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:55.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:55.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:56.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:56.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:56.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:56.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:56.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:56.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:56.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:56.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:56.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:56.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:08:56.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:08:56.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:56.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:56.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:57.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:57.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:57.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:57.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:57.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:57.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:57.095 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:57.095 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:57.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:57.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:57.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:57.438 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:57.964 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:57.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:57.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:57.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:57.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:57.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:57.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:57.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:57.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:08:58.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:08:58.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:08:58.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:08:58.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:08:58.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:08:58.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:08:58.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:08:58.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:08:58.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:08:58.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:08:58.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:08:58.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:08:58.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:08:58.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:08:58.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:08:58.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:08:58.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:08:59.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:08:59.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:08:59.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:08:59.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:08:59.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:08:59.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:08:59.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:08:59.395 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:08:59.396 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:09:00.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:09:00.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:09:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:09:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:09:00.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:09:00.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:09:00.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:00.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:00.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:00.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:00.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:00.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:00.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:00.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:09:00.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:00.617 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:00.618 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:01.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:09:01.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:09:01.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:09:01.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:09:01.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:09:01.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:09:01.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:09:01.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:09:01.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:09:01.545 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=6 2021-07-26T07:09:01.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 6 '!=' null 2021-07-26T07:09:01.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:09:01.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:09:01.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:09:01.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:09:01.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:01.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:01.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:01.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:01.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:01.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:01.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:01.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:09:02.000 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:02.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:02.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:02.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:09:02.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:09:02.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:09:02.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:09:02.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:09:02.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:09:02.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:09:02.458 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:09:02.458 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:09:02.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:09:02.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:09:02.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:09:02.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:09:02.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:09:03.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:09:03.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:03.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:03.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:03.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:03.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:03.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:03.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:03.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:09:03.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:03.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:03.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:03.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:09:03.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:09:03.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:09:03.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:09:03.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:09:03.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:09:03.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:09:03.841 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:09:03.842 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:09:04.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:09:04.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:09:04.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:09:04.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:09:04.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:09:04.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:09:04.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:09:04.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:09:04.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:09:04.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:09:04.415 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:09:04.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:09:04.416 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:09:04.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:09:04.795 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:09:04.795 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:09:04.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:09:05.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:09:05.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:09:05.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:09:05.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2021-07-26T07:09:05.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-07-26T07:09:05.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-07-26T07:09:05.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2021-07-26T07:09:05.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2021-07-26T07:09:05.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2021-07-26T07:09:05.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-07-26T07:09:05.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2021-07-26T07:09:05.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-07-26T07:09:05.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-07-26T07:09:05.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-07-26T07:09:05.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2021-07-26T07:09:05.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-07-26T07:09:05.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2021-07-26T07:09:05.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2021-07-26T07:09:05.265 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-07-26T07:09:05.265 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:09:05.266 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-07-26T07:09:05.266 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-07-26T07:09:05.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2021-07-26T07:09:05.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-07-26T07:09:05.282 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:09:05.283 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:09:05.283 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-07-26T07:09:05.283 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2021-07-26T07:09:05.284 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-07-26T07:09:05.284 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:09:05.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2021-07-26T07:09:05.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2021-07-26T07:09:05.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-07-26T07:09:05.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:09:05.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-07-26T07:09:05.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:09:05.293 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:09:05.294 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:09:05.294 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-07-26T07:09:05.294 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:09:05.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-07-26T07:09:05.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-07-26T07:09:05.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:09:05.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:09:05.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:09:05.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:09:05.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-07-26T07:09:05.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:09:05.313 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:09:05.313 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2021-07-26T07:09:05.313 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-07-26T07:09:05.314 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-07-26T07:09:05.314 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:09:05.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2021-07-26T07:09:05.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2021-07-26T07:09:05.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2021-07-26T07:09:05.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:09:05.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2021-07-26T07:09:05.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:09:05.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:09:05.326 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:09:05.326 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-07-26T07:09:05.326 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:09:05.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-07-26T07:09:05.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-07-26T07:09:05.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:09:05.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:09:05.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:09:05.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:09:05.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-07-26T07:09:05.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2021-07-26T07:09:05.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T07:09:05.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T07:09:05.835 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T07:09:05.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2021-07-26T07:09:05.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:09:05.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:09:05.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:09:05.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:09:05.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:09:05.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:09:05.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-07-26T07:09:05.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:09:05.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:09:05.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:09:05.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:09:05.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:09:05.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:09:05.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:09:05.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:09:05.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:09:05.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:09:05.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:09:05.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:09:05.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:09:05.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:09:05.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:09:05.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:09:05.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:09:05.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:09:05.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:09:05.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:09:05.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:09:05.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:09:06.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:09:06.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:06.008 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:06.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:09:06.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:09:06.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:09:06.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-07-26T07:09:06.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-07-26T07:09:06.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-07-26T07:09:06.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-07-26T07:09:06.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:09:06.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:09:06.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:09:06.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:09:06.013 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:09:06.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:09:06.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:09:06.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:09:06.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:09:06.014 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:09:06.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:09:06.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:09:06.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:09:06.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:09:06.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:09:06.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:09:06.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:09:06.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:09:06.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:09:06.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:09:06.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:09:06.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:09:06.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:09:06.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:06.025 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:06.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:09:06.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:09:06.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:09:06.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-07-26T07:09:06.028 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:09:06.028 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:06.028 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:06.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257957 2021-07-26T07:09:06.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:09:06.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:09:06.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:09:06.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-07-26T07:09:06.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2021-07-26T07:09:06.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2021-07-26T07:09:06.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2021-07-26T07:09:06.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-07-26T07:09:06.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:09:06.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:09:06.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:09:06.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-07-26T07:09:06.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-07-26T07:09:06.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:09:06.418 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:06.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:06.420 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:06.420 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:06.420 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:06.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:06.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:09:06.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:09:06.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:09:06.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:09:06.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:09:06.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:09:06.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:09:06.728 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:09:06.728 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:09:06.728 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:09:06.729 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:09:06.730 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:06.730 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:06.730 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:09:06.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:09:06.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get fsid 2021-07-26T07:09:06.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:09:06.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:09:06.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:09:06.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:09:06.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:09:06.820 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:09:06.820 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:09:06.821 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:09:06.821 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:09:06.821 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:06.822 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:06.822 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:09:06.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:09:06.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get mon_host 2021-07-26T07:09:06.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2021-07-26T07:09:06.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2021-07-26T07:09:06.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:09:06.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:09:06.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:09:06.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2021-07-26T07:09:06.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:09:07.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:09:07.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:07.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:07.833 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:07.833 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:07.834 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:07.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:07.834 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:09:07.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:09:07.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2021-07-26T07:09:07.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:09:07.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:09:07.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:09:07.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:09:07.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:09:07.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:09:07.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:09:07.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:09:07.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:09:07.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:09:07.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:09:07.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:09:07.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:09:07.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:09:07.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:09:07.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:09:07.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:07.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:07.912 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:07.913 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:07.913 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:07.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:07.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:07.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:09:07.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:09:07.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:09:07.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:09:07.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:09:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:09:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:09:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:09:07.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:09:07.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:09:07.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:09:07.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98bfbe02-650a-4738-aa9f-48a04b8eb62c 2021-07-26T07:09:07.918 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 98bfbe02-650a-4738-aa9f-48a04b8eb62c 2021-07-26T07:09:07.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 98bfbe02-650a-4738-aa9f-48a04b8eb62c' 2021-07-26T07:09:07.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:09:07.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTX/5gKCT2NhAAtd5c+7YYG2NPYnndjRzxvA== 2021-07-26T07:09:07.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTX/5gKCT2NhAAtd5c+7YYG2NPYnndjRzxvA=="}' 2021-07-26T07:09:07.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98bfbe02-650a-4738-aa9f-48a04b8eb62c -i td/osd-recovery-stats/0/new.json 2021-07-26T07:09:08.275 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:09:08.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2021-07-26T07:09:08.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCTX/5gKCT2NhAAtd5c+7YYG2NPYnndjRzxvA== --osd-uuid 98bfbe02-650a-4738-aa9f-48a04b8eb62c 2021-07-26T07:09:08.318 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:08.306+0000 7f25f2bedf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:08.337 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:08.326+0000 7f25f2bedf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:08.338 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:08.326+0000 7f25f2bedf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:08.339 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:08.326+0000 7f25f2bedf40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:09:08.339 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:08.326+0000 7f25f2bedf40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:09:08.340 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:08.326+0000 7f25f2bedf40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:09:08.383 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:08.370+0000 7f25f2bedf40 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-07-26T07:09:14.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-07-26T07:09:14.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:09:14.224 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:09:14.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:09:14.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:09:14.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:09:14.725 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:09:14.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:14.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:09:14.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:09:14.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:09:14.748 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:14.734+0000 7f6189b35f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:14.751 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:14.738+0000 7f6189b35f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:14.752 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:14.738+0000 7f6189b35f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:15.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:09:15.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:09:15.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:09:15.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:09:15.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:09:15.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:15.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:09:15.082 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:09:15.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:15.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:15.291 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:15.278+0000 7f6189b35f40 -1 Falling back to public interface 2021-07-26T07:09:15.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:16.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:16.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:16.422 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:09:16.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:09:16.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:16.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:16.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:17.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:17.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:17.767 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:09:17.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:09:17.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:17.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:17.857 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:17.846+0000 7f6189b35f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:09:18.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:19.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:19.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:19.147 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:09:19.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:09:19.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:19.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:09:19.587 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/274894,v1:127.0.0.1:6803/274894] [v2:127.0.0.1:6804/274894,v1:127.0.0.1:6805/274894] exists,up 98bfbe02-650a-4738-aa9f-48a04b8eb62c 2021-07-26T07:09:19.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:09:19.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:09:19.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:09:19.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2021-07-26T07:09:19.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:09:19.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:09:19.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:09:19.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:09:19.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:09:19.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:09:19.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:09:19.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:09:19.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:09:19.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:09:19.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:09:19.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:09:19.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:09:19.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:09:19.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:09:19.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:09:19.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:19.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:19.596 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:19.596 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:19.597 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:19.597 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:19.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:19.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:09:19.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:09:19.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:09:19.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:09:19.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:09:19.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:09:19.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:09:19.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:19.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:09:19.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:09:19.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:09:19.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:09:19.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=644991dd-d097-4275-a702-b1f63cb3280c 2021-07-26T07:09:19.601 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 644991dd-d097-4275-a702-b1f63cb3280c 2021-07-26T07:09:19.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 644991dd-d097-4275-a702-b1f63cb3280c' 2021-07-26T07:09:19.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:09:19.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfX/5g54DyIxAAvHc/p1UF5PTuQRKggVfgeA== 2021-07-26T07:09:19.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfX/5g54DyIxAAvHc/p1UF5PTuQRKggVfgeA=="}' 2021-07-26T07:09:19.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 644991dd-d097-4275-a702-b1f63cb3280c -i td/osd-recovery-stats/1/new.json 2021-07-26T07:09:20.091 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:09:20.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2021-07-26T07:09:20.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCfX/5g54DyIxAAvHc/p1UF5PTuQRKggVfgeA== --osd-uuid 644991dd-d097-4275-a702-b1f63cb3280c 2021-07-26T07:09:20.140 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:20.126+0000 7f685dab5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:20.160 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:20.146+0000 7f685dab5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:20.162 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:20.150+0000 7f685dab5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:20.162 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:20.150+0000 7f685dab5f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:09:20.163 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:20.150+0000 7f685dab5f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:09:20.163 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:20.150+0000 7f685dab5f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:09:20.204 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:20.190+0000 7f685dab5f40 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-07-26T07:09:26.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-07-26T07:09:26.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:09:26.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:09:26.022 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:09:26.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:09:26.506 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:09:26.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:09:26.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:26.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:09:26.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:09:26.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:09:26.534 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:26.522+0000 7fb9bad83f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:26.536 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:26.522+0000 7fb9bad83f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:26.537 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:26.526+0000 7fb9bad83f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:26.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:09:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:09:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:09:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:09:26.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:09:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:26.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:09:26.881 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:09:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:26.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:27.067 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:27.054+0000 7fb9bad83f40 -1 Falling back to public interface 2021-07-26T07:09:27.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:28.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:28.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:28.260 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:09:28.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:09:28.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:28.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:28.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:29.572 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:29.558+0000 7fb9bad83f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:09:29.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:29.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:29.616 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:09:29.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:09:29.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:29.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:30.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:31.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:31.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:31.207 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:09:31.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:09:31.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:31.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:09:31.567 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/276300,v1:127.0.0.1:6811/276300] [v2:127.0.0.1:6812/276300,v1:127.0.0.1:6813/276300] exists,up 644991dd-d097-4275-a702-b1f63cb3280c 2021-07-26T07:09:31.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:09:31.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:09:31.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:09:31.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2021-07-26T07:09:31.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:09:31.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:09:31.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:09:31.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:09:31.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2021-07-26T07:09:31.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:09:31.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:09:31.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:09:31.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:09:31.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-07-26T07:09:31.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-07-26T07:09:31.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:09:31.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:09:31.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:09:31.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:09:31.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:09:31.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:31.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:31.574 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:31.575 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:31.575 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:31.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:31.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:31.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:09:31.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:09:31.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:09:31.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:09:31.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:09:31.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:09:31.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:09:31.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:31.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:09:31.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:09:31.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2021-07-26T07:09:31.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:09:31.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d469865-e5e7-4e2f-b7a5-506e42666f59 2021-07-26T07:09:31.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2d469865-e5e7-4e2f-b7a5-506e42666f59' 2021-07-26T07:09:31.580 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 2d469865-e5e7-4e2f-b7a5-506e42666f59 2021-07-26T07:09:31.581 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:09:31.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrX/5gFeSkIhAAt3RQMHLnbw5Qwx7yZQapSw== 2021-07-26T07:09:31.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrX/5gFeSkIhAAt3RQMHLnbw5Qwx7yZQapSw=="}' 2021-07-26T07:09:31.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d469865-e5e7-4e2f-b7a5-506e42666f59 -i td/osd-recovery-stats/2/new.json 2021-07-26T07:09:32.109 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:09:32.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2021-07-26T07:09:32.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCrX/5gFeSkIhAAt3RQMHLnbw5Qwx7yZQapSw== --osd-uuid 2d469865-e5e7-4e2f-b7a5-506e42666f59 2021-07-26T07:09:32.155 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:32.142+0000 7f5f7eb9bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:32.162 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:32.150+0000 7f5f7eb9bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:32.164 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:32.150+0000 7f5f7eb9bf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:32.165 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:32.154+0000 7f5f7eb9bf40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:09:32.166 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:32.154+0000 7f5f7eb9bf40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:09:32.166 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:32.154+0000 7f5f7eb9bf40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:09:32.208 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:32.194+0000 7f5f7eb9bf40 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-07-26T07:09:38.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-07-26T07:09:38.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:09:38.388 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T07:09:38.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:09:38.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:09:38.884 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T07:09:38.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:09:38.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:38.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:09:38.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:09:38.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:09:38.912 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:38.897+0000 7fc581c65f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:38.914 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:38.901+0000 7fc581c65f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:38.921 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:38.901+0000 7fc581c65f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:39.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:09:39.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:09:39.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:09:39.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:09:39.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:09:39.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:39.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:09:39.232 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:09:39.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:09:39.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:39.459 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:39.445+0000 7fc581c65f40 -1 Falling back to public interface 2021-07-26T07:09:39.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:40.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:40.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:40.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:09:40.606 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:09:40.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:40.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:09:40.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:41.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:41.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:41.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:09:41.950 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:09:41.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:09:41.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:42.174 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:42.161+0000 7fc581c65f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:09:42.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:43.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:43.400 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:43.400 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:09:43.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:09:43.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:43.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:09:43.768 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/277707,v1:127.0.0.1:6819/277707] [v2:127.0.0.1:6820/277707,v1:127.0.0.1:6821/277707] exists,up 2d469865-e5e7-4e2f-b7a5-506e42666f59 2021-07-26T07:09:43.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:09:43.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:09:43.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:09:43.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2021-07-26T07:09:43.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:09:43.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:09:43.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T07:09:43.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:09:43.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2021-07-26T07:09:43.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:09:43.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:09:43.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:09:43.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:09:43.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-07-26T07:09:43.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-07-26T07:09:43.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:09:43.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:09:43.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:09:43.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:09:43.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:09:43.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:09:43.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:09:43.778 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:09:43.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:09:43.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:09:43.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:43.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:09:43.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:09:43.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:09:43.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:09:43.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:09:43.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:09:43.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:09:43.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:09:43.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:43.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:09:43.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:09:43.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2021-07-26T07:09:43.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:09:43.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0acd3eb-409b-4447-82fc-d91d326a327d 2021-07-26T07:09:43.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c0acd3eb-409b-4447-82fc-d91d326a327d' 2021-07-26T07:09:43.787 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 c0acd3eb-409b-4447-82fc-d91d326a327d 2021-07-26T07:09:43.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:09:43.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3X/5g5lrDLhAAbD1jnQbT9Lvn46yE/kG6Bg== 2021-07-26T07:09:43.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3X/5g5lrDLhAAbD1jnQbT9Lvn46yE/kG6Bg=="}' 2021-07-26T07:09:43.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0acd3eb-409b-4447-82fc-d91d326a327d -i td/osd-recovery-stats/3/new.json 2021-07-26T07:09:44.462 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:09:44.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2021-07-26T07:09:44.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC3X/5g5lrDLhAAbD1jnQbT9Lvn46yE/kG6Bg== --osd-uuid c0acd3eb-409b-4447-82fc-d91d326a327d 2021-07-26T07:09:44.507 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:44.493+0000 7f41302bbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:44.527 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:44.513+0000 7f41302bbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:44.529 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:44.517+0000 7f41302bbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:44.530 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:44.517+0000 7f41302bbf40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:09:44.530 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:44.517+0000 7f41302bbf40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:09:44.530 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:44.517+0000 7f41302bbf40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:09:44.597 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:44.585+0000 7f41302bbf40 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-07-26T07:09:50.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-07-26T07:09:50.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:09:50.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T07:09:50.817 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T07:09:50.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:09:51.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T07:09:51.322 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T07:09:51.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:09:51.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:09:51.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:09:51.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:09:51.351 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:51.337+0000 7ff5dbab3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:51.377 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:51.365+0000 7ff5dbab3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:51.378 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:51.365+0000 7ff5dbab3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:09:51.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T07:09:51.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:09:51.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:09:51.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:09:51.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:09:51.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:51.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:09:51.725 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:09:51.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:51.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:09:51.911 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:51.897+0000 7ff5dbab3f40 -1 Falling back to public interface 2021-07-26T07:09:52.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:53.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:53.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:53.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:09:53.076 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:09:53.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:53.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:09:53.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:54.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:54.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:54.419 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:09:54.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:09:54.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:54.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:09:54.567 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:09:54.553+0000 7ff5dbab3f40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:09:54.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:09:55.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:09:55.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:09:55.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:09:55.886 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:09:55.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:09:55.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:09:56.294 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/279113,v1:127.0.0.1:6827/279113] [v2:127.0.0.1:6828/279113,v1:127.0.0.1:6829/279113] exists,up c0acd3eb-409b-4447-82fc-d91d326a327d 2021-07-26T07:09:56.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:09:56.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:09:56.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:09:56.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-07-26T07:09:56.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2021-07-26T07:09:56.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-07-26T07:09:57.687 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T07:09:57.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:09:58.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2021-07-26T07:09:59.430 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 min_size to 2 2021-07-26T07:09:59.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2021-07-26T07:09:59.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:09:59.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:09:59.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:09:59.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:09:59.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:09:59.447 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:09:59.447 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:09:59.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:09:59.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:09:59.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:09:59.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:09:59.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:09:59.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:09:59.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:09:59.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:10:00.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:10:00.039 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:10:00.039 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:10:00.039 INFO:tasks.workunit.client.0.smithi094.stderr:3' 2021-07-26T07:10:00.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:10:00.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:00.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:10:00.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836491 2021-07-26T07:10:00.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836491 2021-07-26T07:10:00.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491' 2021-07-26T07:10:00.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:00.183 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:10:00.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672968 2021-07-26T07:10:00.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672968 2021-07-26T07:10:00.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672968' 2021-07-26T07:10:00.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:00.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:10:00.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509446 2021-07-26T07:10:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509446 2021-07-26T07:10:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672968 2-64424509446' 2021-07-26T07:10:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:00.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:10:00.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345923 2021-07-26T07:10:00.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345923 2021-07-26T07:10:00.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836491 1-42949672968 2-64424509446 3-85899345923' 2021-07-26T07:10:00.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:00.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836491 2021-07-26T07:10:00.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:00.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:10:00.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836491 2021-07-26T07:10:00.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:00.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836491 2021-07-26T07:10:00.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-07-26T07:10:00.613 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836491 2021-07-26T07:10:00.613 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:00.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836489 -lt 21474836491 2021-07-26T07:10:00.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:10:01.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:10:01.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:10:02.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836491 -lt 21474836491 2021-07-26T07:10:02.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:02.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672968 2021-07-26T07:10:02.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:02.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:10:02.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672968 2021-07-26T07:10:02.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:02.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672968 2021-07-26T07:10:02.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-07-26T07:10:02.378 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672968 2021-07-26T07:10:02.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:10:02.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672969 -lt 42949672968 2021-07-26T07:10:02.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:02.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509446 2021-07-26T07:10:02.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:02.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:10:02.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509446 2021-07-26T07:10:02.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:02.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509446 2021-07-26T07:10:02.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-07-26T07:10:02.783 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509446 2021-07-26T07:10:02.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:10:03.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509446 -lt 64424509446 2021-07-26T07:10:03.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:03.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345923 2021-07-26T07:10:03.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:03.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:10:03.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345923 2021-07-26T07:10:03.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:03.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345923 2021-07-26T07:10:03.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2021-07-26T07:10:03.191 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345923 2021-07-26T07:10:03.191 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:10:03.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345924 -lt 85899345923 2021-07-26T07:10:03.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:10:03.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:03.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:04.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:10:04.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:04.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:04.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:04.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:04.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:04.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:04.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:04.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:10:04.402 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:04.403 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:04.403 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:04.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:10:04.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:10:04.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:10:04.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2021-07-26T07:10:04.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:04.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2021-07-26T07:10:05.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:05.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2021-07-26T07:10:05.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:05.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2021-07-26T07:10:05.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:05.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2021-07-26T07:10:05.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:05.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2021-07-26T07:10:05.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:05.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2021-07-26T07:10:06.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:06.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2021-07-26T07:10:06.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:06.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2021-07-26T07:10:06.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:06.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2021-07-26T07:10:06.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:06.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2021-07-26T07:10:06.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:06.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2021-07-26T07:10:06.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:06.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2021-07-26T07:10:07.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:07.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2021-07-26T07:10:07.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:07.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2021-07-26T07:10:07.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:07.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2021-07-26T07:10:07.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:07.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2021-07-26T07:10:07.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:07.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2021-07-26T07:10:08.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:08.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2021-07-26T07:10:08.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:08.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2021-07-26T07:10:08.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:08.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2021-07-26T07:10:08.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:08.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2021-07-26T07:10:08.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:08.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2021-07-26T07:10:09.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:09.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2021-07-26T07:10:09.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:09.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2021-07-26T07:10:09.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:09.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2021-07-26T07:10:09.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:09.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2021-07-26T07:10:09.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:09.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2021-07-26T07:10:10.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:10.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2021-07-26T07:10:10.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:10.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2021-07-26T07:10:10.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:10.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2021-07-26T07:10:10.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:10.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2021-07-26T07:10:10.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:10.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2021-07-26T07:10:10.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:10.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2021-07-26T07:10:11.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:11.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2021-07-26T07:10:11.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:11.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2021-07-26T07:10:11.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:11.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2021-07-26T07:10:11.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:11.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2021-07-26T07:10:11.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:11.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2021-07-26T07:10:11.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:11.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2021-07-26T07:10:12.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:12.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2021-07-26T07:10:12.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:12.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2021-07-26T07:10:12.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:12.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2021-07-26T07:10:12.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:12.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2021-07-26T07:10:12.957 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:12.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2021-07-26T07:10:13.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:13.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2021-07-26T07:10:13.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:13.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2021-07-26T07:10:13.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:13.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2021-07-26T07:10:13.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:13.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2021-07-26T07:10:13.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:13.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2021-07-26T07:10:14.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:14.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2021-07-26T07:10:14.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:14.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2021-07-26T07:10:14.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:14.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2021-07-26T07:10:14.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:14.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2021-07-26T07:10:14.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:14.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2021-07-26T07:10:14.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:14.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2021-07-26T07:10:15.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:15.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2021-07-26T07:10:15.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:15.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2021-07-26T07:10:15.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:15.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2021-07-26T07:10:15.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:15.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2021-07-26T07:10:15.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:15.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2021-07-26T07:10:16.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:16.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2021-07-26T07:10:16.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:16.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2021-07-26T07:10:16.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:16.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2021-07-26T07:10:16.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:16.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2021-07-26T07:10:16.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:16.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2021-07-26T07:10:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:16.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2021-07-26T07:10:17.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:17.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2021-07-26T07:10:17.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:17.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2021-07-26T07:10:17.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:17.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2021-07-26T07:10:17.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:17.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2021-07-26T07:10:17.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:17.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2021-07-26T07:10:18.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:18.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2021-07-26T07:10:18.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:18.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2021-07-26T07:10:18.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:18.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2021-07-26T07:10:18.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:18.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2021-07-26T07:10:18.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:18.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2021-07-26T07:10:18.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:18.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2021-07-26T07:10:19.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:19.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2021-07-26T07:10:19.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:19.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2021-07-26T07:10:19.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:19.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2021-07-26T07:10:19.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:19.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2021-07-26T07:10:19.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:19.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2021-07-26T07:10:20.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:20.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2021-07-26T07:10:20.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:20.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2021-07-26T07:10:20.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:20.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2021-07-26T07:10:20.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:20.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2021-07-26T07:10:20.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:20.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2021-07-26T07:10:21.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:21.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2021-07-26T07:10:21.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:21.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2021-07-26T07:10:21.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:21.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2021-07-26T07:10:21.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:21.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2021-07-26T07:10:21.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:21.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2021-07-26T07:10:21.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:21.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2021-07-26T07:10:22.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:22.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2021-07-26T07:10:22.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:22.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2021-07-26T07:10:22.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:22.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2021-07-26T07:10:22.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:22.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2021-07-26T07:10:22.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:22.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2021-07-26T07:10:23.041 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:23.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2021-07-26T07:10:23.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:23.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2021-07-26T07:10:23.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:23.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2021-07-26T07:10:23.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:23.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2021-07-26T07:10:23.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:23.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2021-07-26T07:10:23.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:23.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2021-07-26T07:10:24.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:24.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2021-07-26T07:10:24.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:24.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2021-07-26T07:10:24.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:24.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2021-07-26T07:10:24.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:24.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2021-07-26T07:10:24.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:24.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2021-07-26T07:10:25.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:25.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2021-07-26T07:10:25.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:25.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2021-07-26T07:10:25.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:25.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2021-07-26T07:10:25.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:25.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2021-07-26T07:10:25.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:25.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2021-07-26T07:10:25.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:25.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2021-07-26T07:10:26.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:26.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2021-07-26T07:10:26.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:26.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2021-07-26T07:10:26.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:26.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2021-07-26T07:10:26.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:26.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2021-07-26T07:10:26.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2021-07-26T07:10:27.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:27.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2021-07-26T07:10:27.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:27.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2021-07-26T07:10:27.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:27.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2021-07-26T07:10:27.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:27.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2021-07-26T07:10:27.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:27.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2021-07-26T07:10:28.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:28.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2021-07-26T07:10:28.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:28.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2021-07-26T07:10:28.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:28.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2021-07-26T07:10:28.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:28.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2021-07-26T07:10:28.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:28.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2021-07-26T07:10:28.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:28.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2021-07-26T07:10:29.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:29.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2021-07-26T07:10:29.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:29.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2021-07-26T07:10:29.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:29.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2021-07-26T07:10:29.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:29.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2021-07-26T07:10:29.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:29.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2021-07-26T07:10:30.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:30.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2021-07-26T07:10:30.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:30.339 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2021-07-26T07:10:30.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:30.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2021-07-26T07:10:30.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:30.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2021-07-26T07:10:30.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:30.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2021-07-26T07:10:31.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:31.063 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2021-07-26T07:10:31.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:31.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2021-07-26T07:10:31.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:31.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2021-07-26T07:10:31.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:31.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2021-07-26T07:10:31.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:31.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2021-07-26T07:10:32.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:32.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2021-07-26T07:10:32.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:32.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2021-07-26T07:10:32.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:32.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2021-07-26T07:10:32.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:32.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2021-07-26T07:10:32.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:32.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2021-07-26T07:10:32.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:32.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2021-07-26T07:10:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2021-07-26T07:10:33.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:33.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2021-07-26T07:10:33.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:33.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2021-07-26T07:10:33.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:33.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2021-07-26T07:10:33.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:33.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2021-07-26T07:10:34.210 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:34.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2021-07-26T07:10:34.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:34.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2021-07-26T07:10:34.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:34.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2021-07-26T07:10:34.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:34.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2021-07-26T07:10:34.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:34.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2021-07-26T07:10:35.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:35.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2021-07-26T07:10:35.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:35.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2021-07-26T07:10:35.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:35.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2021-07-26T07:10:35.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:35.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2021-07-26T07:10:35.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:35.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2021-07-26T07:10:36.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:36.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2021-07-26T07:10:36.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:36.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2021-07-26T07:10:36.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:36.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2021-07-26T07:10:36.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:36.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2021-07-26T07:10:36.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:36.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2021-07-26T07:10:37.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:37.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2021-07-26T07:10:37.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:37.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2021-07-26T07:10:37.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:37.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2021-07-26T07:10:37.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:37.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2021-07-26T07:10:37.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:37.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2021-07-26T07:10:37.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:37.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2021-07-26T07:10:38.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:38.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2021-07-26T07:10:38.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:38.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2021-07-26T07:10:38.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:38.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2021-07-26T07:10:38.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:38.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2021-07-26T07:10:38.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:38.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2021-07-26T07:10:39.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:39.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2021-07-26T07:10:39.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:39.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2021-07-26T07:10:39.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:39.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2021-07-26T07:10:39.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:39.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2021-07-26T07:10:39.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:39.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2021-07-26T07:10:40.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:40.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2021-07-26T07:10:40.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:40.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2021-07-26T07:10:40.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:40.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2021-07-26T07:10:40.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:40.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2021-07-26T07:10:40.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:40.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2021-07-26T07:10:41.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:41.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2021-07-26T07:10:41.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:41.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2021-07-26T07:10:41.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:41.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2021-07-26T07:10:41.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:41.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2021-07-26T07:10:41.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:41.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2021-07-26T07:10:41.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:41.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2021-07-26T07:10:42.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-07-26T07:10:42.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2021-07-26T07:10:42.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2021-07-26T07:10:42.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:10:42.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:10:42.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:10:42.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:10:42.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2021-07-26T07:10:42.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2021-07-26T07:10:42.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T07:10:42.790 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:10:42.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test obj1 2021-07-26T07:10:42.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:10:43.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2021-07-26T07:10:43.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2021-07-26T07:10:43.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test 2021-07-26T07:10:43.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T07:10:43.194 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test obj1 2021-07-26T07:10:43.194 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:10:43.195 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:10:43.195 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:10:43.195 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:10:43.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:10:43.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test obj1 2021-07-26T07:10:43.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:10:43.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2021-07-26T07:10:43.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2021-07-26T07:10:44.791 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is set 2021-07-26T07:10:44.804 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2021-07-26T07:10:44.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 274894 2021-07-26T07:10:44.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2021-07-26T07:10:45.248 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T07:10:45.256 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2021-07-26T07:10:45.865 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T07:10:45.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2021-07-26T07:10:45.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2021-07-26T07:10:45.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2021-07-26T07:10:45.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2021-07-26T07:10:45.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2021-07-26T07:10:45.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2021-07-26T07:10:45.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2021-07-26T07:10:45.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2021-07-26T07:10:45.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2021-07-26T07:10:45.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2021-07-26T07:10:45.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2021-07-26T07:10:46.476 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already out. 2021-07-26T07:10:46.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2021-07-26T07:10:46.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2021-07-26T07:10:47.715 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is unset 2021-07-26T07:10:47.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2021-07-26T07:10:47.729 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:10:47.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:10:47.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:10:47.731 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:10:48.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2021-07-26T07:10:48.390 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T07:10:48.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2021-07-26T07:10:50.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2021-07-26T07:10:50.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:10:50.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:10:50.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:10:50.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:10:50.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:10:50.406 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:10:50.406 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:10:50.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:10:50.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:10:50.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:10:50.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:10:50.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:10:50.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:10:50.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:10:50.620 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:10:50.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:10:50.989 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:10:50.990 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:10:50.990 INFO:tasks.workunit.client.0.smithi094.stderr:3' 2021-07-26T07:10:50.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:10:50.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:50.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:10:51.084 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-07-26T07:10:51.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T07:10:51.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T07:10:51.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T07:10:51.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:51.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:10:51.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672981 2021-07-26T07:10:51.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672981 2021-07-26T07:10:51.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672981' 2021-07-26T07:10:51.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:51.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:10:51.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509458 2021-07-26T07:10:51.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509458 2021-07-26T07:10:51.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672981 2-64424509458' 2021-07-26T07:10:51.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:10:51.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:10:51.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345936 2021-07-26T07:10:51.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345936 2021-07-26T07:10:51.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672981 2-64424509458 3-85899345936' 2021-07-26T07:10:51.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:51.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672981 2021-07-26T07:10:51.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:51.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:10:51.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672981 2021-07-26T07:10:51.606 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:51.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672981 2021-07-26T07:10:51.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2021-07-26T07:10:51.607 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672981 2021-07-26T07:10:51.608 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:10:51.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672978 -lt 42949672981 2021-07-26T07:10:51.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:10:52.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:10:52.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:10:53.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672981 -lt 42949672981 2021-07-26T07:10:53.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:53.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509458 2021-07-26T07:10:53.357 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:53.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:10:53.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:53.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509458 2021-07-26T07:10:53.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509458 2021-07-26T07:10:53.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2021-07-26T07:10:53.362 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509458 2021-07-26T07:10:53.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:10:53.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509458 -lt 64424509458 2021-07-26T07:10:53.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:10:53.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345936 2021-07-26T07:10:53.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:10:53.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:10:53.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345936 2021-07-26T07:10:53.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:10:53.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345936 2021-07-26T07:10:53.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2021-07-26T07:10:53.727 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345936 2021-07-26T07:10:53.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:10:54.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345936 -lt 85899345936 2021-07-26T07:10:54.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:10:54.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:54.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:54.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:10:54.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:54.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:54.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:54.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:54.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:54.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:54.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:54.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:10:54.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:54.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:54.967 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:55.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:10:55.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:10:55.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:10:55.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:10:55.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:10:55.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:10:55.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:10:55.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:55.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:55.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:55.557 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:55.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:55.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:55.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:55.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:10:55.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:55.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:55.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:56.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:10:56.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:10:56.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:10:56.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:10:56.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:10:56.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:10:56.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:10:56.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:10:56.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:10:56.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:10:56.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:10:56.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:10:56.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:10:56.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:10:56.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:10:56.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:56.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:56.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:56.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:56.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:56.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:56.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:57.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:10:57.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:57.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:57.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:57.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:10:57.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:10:57.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:10:57.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:10:57.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:10:57.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:10:57.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:10:57.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:10:57.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:10:58.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:10:58.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:10:58.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:10:58.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:10:58.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:10:58.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:10:58.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:58.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:58.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:58.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:58.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:58.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:58.260 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:10:58.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:10:58.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:10:58.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:10:58.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:10:59.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:10:59.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:10:59.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:10:59.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:10:59.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:10:59.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:10:59.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:10:59.066 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:10:59.067 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:10:59.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:10:59.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:10:59.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:10:59.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:10:59.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:10:59.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:10:59.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:10:59.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:10:59.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:10:59.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:10:59.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:10:59.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:10:59.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:00.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:00.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:00.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:00.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:00.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:00.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:00.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:00.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:00.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:00.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:00.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:00.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:00.549 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:01.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:11:01.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:11:01.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:01.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:01.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:01.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:01.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:01.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:01.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:01.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:01.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:01.180 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:01.181 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:01.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:01.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:01.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:01.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:01.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:01.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:01.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:01.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:01.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:01.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:01.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:01.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:01.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:02.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=6 2021-07-26T07:11:02.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 6 '!=' null 2021-07-26T07:11:02.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:02.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:02.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:02.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:02.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:02.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:02.649 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:02.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:02.650 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:02.651 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:02.651 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:03.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:03.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:03.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:03.034 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:03.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:03.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:03.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:03.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:03.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:03.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:03.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:03.462 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:03.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:03.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=6 2021-07-26T07:11:03.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 6 '!=' null 2021-07-26T07:11:03.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:03.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:03.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:04.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:04.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:04.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:04.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:04.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:04.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:04.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:04.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:04.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:04.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:04.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:04.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:04.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:04.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:04.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:04.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:04.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:04.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:04.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:04.905 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:04.905 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:05.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:05.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:05.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:05.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:05.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:05.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:05.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:05.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:05.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:05.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:05.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:05.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:05.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:05.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:05.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:05.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:05.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:06.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:06.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:06.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:06.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:06.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:06.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:06.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:06.310 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:06.311 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:06.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:06.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:06.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:06.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:06.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:06.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:06.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:06.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:06.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:06.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:06.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:06.898 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:06.898 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:07.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:07.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:07.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:07.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:07.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:07.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:07.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:07.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:07.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:07.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:07.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:07.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:07.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:08.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:11:08.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:11:08.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:08.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:08.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:08.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:08.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:08.443 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:08.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:08.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:08.444 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:08.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:08.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:08.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:08.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:08.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:08.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:09.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:09.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:09.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:09.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:09.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:09.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:09.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:09.325 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:09.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:09.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:11:09.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:11:09.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:09.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:09.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:09.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:09.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:09.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:09.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:09.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:09.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:09.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:09.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:10.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:10.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:10.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:10.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:10.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:10.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:10.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:10.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:10.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:10.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:10.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:10.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:10.853 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:11.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:11.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:11.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:11.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:11.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:11.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:11.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:11.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:11.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:11.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:11.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:11.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:11.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:11.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:11.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:11.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:11.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:12.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:12.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:12.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:12.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:12.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:12.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:12.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:12.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:12.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:12.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:11:12.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:11:12.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:12.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:12.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:12.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:12.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:12.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:12.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:12.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:12.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:12.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:12.881 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:13.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:13.237 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:13.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:13.238 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:13.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:13.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:13.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:13.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:13.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:13.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:13.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:13.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:13.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:14.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:14.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:14.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:14.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:14.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:14.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:14.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:14.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:14.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:14.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:14.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:14.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:14.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:14.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:14.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:14.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:14.732 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:15.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:15.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:15.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:15.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:15.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:15.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:15.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:15.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:15.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:15.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:15.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:15.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:15.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:15.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:16.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:16.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:16.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:16.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:16.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:16.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:16.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:16.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:16.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:16.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:16.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:16.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:16.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:16.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:16.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:16.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:16.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:17.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:17.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:17.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:17.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:17.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:17.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:17.495 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:17.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:17.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:17.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:17.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:17.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:17.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:17.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:17.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:17.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:18.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:18.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:18.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:18.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:18.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:18.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:18.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:18.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:18.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:18.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:11:18.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:11:18.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:18.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:18.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:18.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:18.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:18.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:18.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:18.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:18.931 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:18.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:18.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:19.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:19.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:19.268 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:19.269 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:19.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:19.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:19.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:19.743 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:19.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:20.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:20.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:20.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:20.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:20.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:20.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:20.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:20.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:20.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:20.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:20.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:20.446 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:20.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:20.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:20.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:20.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:20.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:21.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:21.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:21.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:21.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:21.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:21.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:21.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:21.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:21.358 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:22.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:11:22.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:11:22.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:22.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:22.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:22.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:22.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:22.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:22.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:22.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:22.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:22.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:22.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:22.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:22.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:22.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:22.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:23.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:23.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:23.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:23.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:23.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:23.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:23.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:23.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:23.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:23.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:11:23.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:11:23.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:23.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:23.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:23.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:23.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:23.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:23.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:23.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:23.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:23.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:23.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:23.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:23.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:23.980 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:23.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:24.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:24.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:24.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:24.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:24.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:24.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:24.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:24.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:24.502 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:25.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:11:25.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:11:25.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:25.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:25.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:25.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:25.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:25.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:25.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:25.140 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:25.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:25.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:25.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:25.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:25.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:25.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:25.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:25.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:25.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:25.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:25.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:25.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:25.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:25.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:25.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:25.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:26.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:11:26.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:11:26.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:26.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:26.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:26.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:26.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:26.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:26.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:26.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:26.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:26.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:26.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:26.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:11:26.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:26.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:26.841 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:27.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:11:27.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:11:27.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:11:27.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:11:27.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:11:27.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:11:27.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:11:27.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:11:27.270 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:11:27.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:11:27.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:11:27.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:11:27.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:11:27.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:11:27.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:11:27.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:11:27.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:11:27.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:11:27.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:11:27.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:11:27.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:11:27.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:11:28.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:11:28.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:11:28.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:11:28.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:11:28.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:11:28.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:11:28.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:11:28.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2021-07-26T07:11:28.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2021-07-26T07:11:28.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-07-26T07:11:28.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-07-26T07:11:28.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2021-07-26T07:11:28.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2021-07-26T07:11:28.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2021-07-26T07:11:28.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-07-26T07:11:28.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2021-07-26T07:11:28.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2021-07-26T07:11:28.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-07-26T07:11:28.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-07-26T07:11:28.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2021-07-26T07:11:28.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-07-26T07:11:28.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2021-07-26T07:11:28.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2021-07-26T07:11:28.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-07-26T07:11:28.725 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:11:28.725 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-07-26T07:11:28.726 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-07-26T07:11:28.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2021-07-26T07:11:28.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-07-26T07:11:28.744 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:11:28.745 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:11:28.745 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2021-07-26T07:11:28.746 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-07-26T07:11:28.746 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-07-26T07:11:28.747 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:11:28.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2021-07-26T07:11:28.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2021-07-26T07:11:28.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-07-26T07:11:28.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:11:28.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-07-26T07:11:28.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:11:28.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:11:28.764 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:11:28.764 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-07-26T07:11:28.765 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:11:28.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-07-26T07:11:28.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-07-26T07:11:28.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:11:28.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:11:28.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:11:28.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:11:28.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-07-26T07:11:28.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:11:28.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:11:28.779 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-07-26T07:11:28.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2021-07-26T07:11:28.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:11:28.780 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-07-26T07:11:28.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2021-07-26T07:11:28.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2021-07-26T07:11:28.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-07-26T07:11:28.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:11:28.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-07-26T07:11:28.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:11:28.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:11:28.792 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:11:28.792 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-07-26T07:11:28.792 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:11:28.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2021-07-26T07:11:28.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2021-07-26T07:11:28.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2021-07-26T07:11:28.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:11:28.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2021-07-26T07:11:28.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:11:28.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-07-26T07:11:28.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2021-07-26T07:11:28.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T07:11:28.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T07:11:29.431 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T07:11:29.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2021-07-26T07:11:29.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:11:29.442 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:11:29.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:11:29.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:11:29.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:11:29.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:11:29.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-07-26T07:11:29.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:11:29.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:11:29.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:11:29.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:11:29.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:11:29.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:11:29.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:11:29.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:11:29.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:11:29.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:11:29.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:11:29.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:11:29.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:11:29.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:11:29.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:11:29.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:11:29.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:11:29.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:11:29.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:11:29.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:11:29.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:11:29.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:11:29.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:11:29.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:29.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:29.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:11:29.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:11:29.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:11:29.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-07-26T07:11:29.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-07-26T07:11:29.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-07-26T07:11:29.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-07-26T07:11:29.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:11:29.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:11:29.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:11:29.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:11:29.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:11:29.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:11:29.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:11:29.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:11:29.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:11:29.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:11:29.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:11:29.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:11:29.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:11:29.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:11:29.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:11:29.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:11:29.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:11:29.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:11:29.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:11:29.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:11:29.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:11:29.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:11:29.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:11:29.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:29.619 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:29.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:11:29.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:11:29.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:11:29.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-07-26T07:11:29.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:11:29.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:29.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:29.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257957 2021-07-26T07:11:29.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:11:29.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:11:29.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:11:29.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-07-26T07:11:29.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2021-07-26T07:11:29.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2021-07-26T07:11:29.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2021-07-26T07:11:29.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2021-07-26T07:11:29.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-07-26T07:11:29.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:11:29.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:11:29.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:11:29.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-07-26T07:11:29.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-07-26T07:11:30.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:11:30.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:30.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:30.018 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:30.019 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:30.019 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:30.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:30.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:11:30.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:11:30.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:11:30.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:11:30.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:11:30.317 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:11:30.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:11:30.319 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:11:30.319 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:11:30.319 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:11:30.320 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:11:30.320 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:30.320 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:30.321 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:11:30.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:11:30.321 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get fsid 2021-07-26T07:11:30.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:11:30.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:11:30.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:11:30.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:11:30.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:11:30.408 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:11:30.408 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:11:30.408 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:11:30.408 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:11:30.409 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:30.409 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:30.409 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:11:30.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:11:30.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get mon_host 2021-07-26T07:11:30.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2021-07-26T07:11:30.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2021-07-26T07:11:30.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:11:30.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:11:30.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:11:30.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2021-07-26T07:11:30.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:11:31.405 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:11:31.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:31.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:31.406 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:31.406 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:31.407 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:31.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:31.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:11:31.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:11:31.440 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2021-07-26T07:11:31.441 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2021-07-26T07:11:31.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:11:31.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2021-07-26T07:11:31.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:11:31.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:11:31.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:11:31.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:11:31.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:11:31.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:11:31.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:11:31.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:11:31.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:11:31.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:11:31.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:11:31.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:11:31.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:11:31.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:11:31.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:11:31.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:11:31.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:31.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:31.450 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:31.450 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:31.451 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:31.451 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:31.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:31.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:11:31.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:11:31.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:11:31.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:11:31.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:11:31.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:11:31.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:11:31.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:31.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:11:31.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:11:31.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:11:31.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:11:31.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1334ce2-e601-44d5-be23-6b49bb85c5a7 2021-07-26T07:11:31.455 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 e1334ce2-e601-44d5-be23-6b49bb85c5a7 2021-07-26T07:11:31.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e1334ce2-e601-44d5-be23-6b49bb85c5a7' 2021-07-26T07:11:31.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:11:31.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjYP5gH643GxAAazpbOEKysE6Lshv3qRzyMQ== 2021-07-26T07:11:31.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjYP5gH643GxAAazpbOEKysE6Lshv3qRzyMQ=="}' 2021-07-26T07:11:31.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1334ce2-e601-44d5-be23-6b49bb85c5a7 -i td/osd-recovery-stats/0/new.json 2021-07-26T07:11:31.835 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:11:31.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2021-07-26T07:11:31.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAjYP5gH643GxAAazpbOEKysE6Lshv3qRzyMQ== --osd-uuid e1334ce2-e601-44d5-be23-6b49bb85c5a7 2021-07-26T07:11:31.881 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:31.867+0000 7fd275cd5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:31.884 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:31.871+0000 7fd275cd5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:31.886 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:31.875+0000 7fd275cd5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:31.887 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:31.875+0000 7fd275cd5f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:11:31.887 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:31.875+0000 7fd275cd5f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:11:31.887 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:31.875+0000 7fd275cd5f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:11:31.968 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:31.955+0000 7fd275cd5f40 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-07-26T07:11:37.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-07-26T07:11:37.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:11:37.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:11:37.625 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:11:37.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:11:38.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:11:38.119 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:11:38.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:38.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:11:38.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:11:38.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:11:38.151 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:38.135+0000 7f94d7c21f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:38.152 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:38.139+0000 7f94d7c21f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:38.152 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:38.139+0000 7f94d7c21f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:38.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:11:38.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:11:38.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:11:38.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:11:38.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:11:38.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:38.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:11:38.484 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:11:38.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:38.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:38.691 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:38.679+0000 7f94d7c21f40 -1 Falling back to public interface 2021-07-26T07:11:38.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:39.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:39.823 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:11:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:11:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:40.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:41.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:41.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:41.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:11:41.192 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:11:41.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:41.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:41.255 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:41.243+0000 7f94d7c21f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:11:41.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:42.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:42.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:42.564 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:11:42.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:11:42.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:42.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:11:42.922 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/287889,v1:127.0.0.1:6803/287889] [v2:127.0.0.1:6804/287889,v1:127.0.0.1:6805/287889] exists,up e1334ce2-e601-44d5-be23-6b49bb85c5a7 2021-07-26T07:11:42.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:11:42.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:11:42.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:11:42.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:11:42.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2021-07-26T07:11:42.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:11:42.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:11:42.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:11:42.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:11:42.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:11:42.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:11:42.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:11:42.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:11:42.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:11:42.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:11:42.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:11:42.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:11:42.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:11:42.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:11:42.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:11:42.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:11:42.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:42.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:42.941 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:42.942 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:42.942 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:42.942 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:42.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:42.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:11:42.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:11:42.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:11:42.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:11:42.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:11:42.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:11:42.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:11:42.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:42.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:11:42.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:11:42.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:11:42.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:11:42.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9e4eea4-9ce1-442c-bd05-1362418efb04 2021-07-26T07:11:42.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b9e4eea4-9ce1-442c-bd05-1362418efb04' 2021-07-26T07:11:42.951 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 b9e4eea4-9ce1-442c-bd05-1362418efb04 2021-07-26T07:11:42.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:11:42.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuYP5g95O1NxAA7Tz7Qsor5bME7XjhZRFZYg== 2021-07-26T07:11:42.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuYP5g95O1NxAA7Tz7Qsor5bME7XjhZRFZYg=="}' 2021-07-26T07:11:42.954 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9e4eea4-9ce1-442c-bd05-1362418efb04 -i td/osd-recovery-stats/1/new.json 2021-07-26T07:11:43.434 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:11:43.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2021-07-26T07:11:43.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAuYP5g95O1NxAA7Tz7Qsor5bME7XjhZRFZYg== --osd-uuid b9e4eea4-9ce1-442c-bd05-1362418efb04 2021-07-26T07:11:43.476 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:43.462+0000 7f242f277f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:43.479 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:43.466+0000 7f242f277f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:43.480 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:43.466+0000 7f242f277f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:43.482 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:43.470+0000 7f242f277f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:11:43.482 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:43.470+0000 7f242f277f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:11:43.482 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:43.470+0000 7f242f277f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:11:43.565 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:43.554+0000 7f242f277f40 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-07-26T07:11:49.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-07-26T07:11:49.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:11:49.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:11:49.617 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:11:49.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:11:50.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:11:50.109 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:11:50.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:11:50.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:11:50.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:50.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:11:50.139 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:50.126+0000 7f8aefe43f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:50.166 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:50.154+0000 7f8aefe43f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:50.167 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:50.154+0000 7f8aefe43f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:50.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:11:50.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:11:50.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:11:50.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:11:50.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:11:50.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:50.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:11:50.448 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:11:50.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:50.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:50.716 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:50.702+0000 7f8aefe43f40 -1 Falling back to public interface 2021-07-26T07:11:50.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:51.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:51.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:51.801 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:11:51.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:11:51.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:51.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:52.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:53.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:53.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:53.168 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:11:53.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:11:53.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:53.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:53.262 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:53.250+0000 7f8aefe43f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:11:53.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:11:54.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:11:54.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:11:54.636 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:11:54.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:11:54.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:11:54.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:11:54.992 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/289296,v1:127.0.0.1:6811/289296] [v2:127.0.0.1:6812/289296,v1:127.0.0.1:6813/289296] exists,up b9e4eea4-9ce1-442c-bd05-1362418efb04 2021-07-26T07:11:54.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:11:54.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:11:54.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:11:54.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:11:54.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2021-07-26T07:11:54.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:11:54.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:11:54.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:11:54.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:11:54.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2021-07-26T07:11:55.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:11:55.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:11:55.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:11:55.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:11:55.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-07-26T07:11:55.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-07-26T07:11:55.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:11:55.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:11:55.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:11:55.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:11:55.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:11:55.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:11:55.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:11:55.002 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:11:55.003 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:11:55.003 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:11:55.003 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:55.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:11:55.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:11:55.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:11:55.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:11:55.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:11:55.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:11:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:11:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:11:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:11:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:11:55.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:11:55.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2021-07-26T07:11:55.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:11:55.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51919392-e5b8-42b8-b6d9-a6fbdeb6dcfd 2021-07-26T07:11:55.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 51919392-e5b8-42b8-b6d9-a6fbdeb6dcfd' 2021-07-26T07:11:55.008 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 51919392-e5b8-42b8-b6d9-a6fbdeb6dcfd 2021-07-26T07:11:55.008 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:11:55.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7YP5gUrxEABAAgceQzRN7hgzAgdVJnyt6VQ== 2021-07-26T07:11:55.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7YP5gUrxEABAAgceQzRN7hgzAgdVJnyt6VQ=="}' 2021-07-26T07:11:55.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51919392-e5b8-42b8-b6d9-a6fbdeb6dcfd -i td/osd-recovery-stats/2/new.json 2021-07-26T07:11:55.494 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:11:55.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2021-07-26T07:11:55.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA7YP5gUrxEABAAgceQzRN7hgzAgdVJnyt6VQ== --osd-uuid 51919392-e5b8-42b8-b6d9-a6fbdeb6dcfd 2021-07-26T07:11:55.540 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:55.526+0000 7f1fcd522f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:55.582 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:55.570+0000 7f1fcd522f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:55.584 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:55.570+0000 7f1fcd522f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:11:55.585 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:55.570+0000 7f1fcd522f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:11:55.585 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:55.570+0000 7f1fcd522f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:11:55.585 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:55.570+0000 7f1fcd522f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:11:55.705 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:11:55.694+0000 7f1fcd522f40 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-07-26T07:12:01.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-07-26T07:12:01.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:12:01.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:12:01.873 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T07:12:01.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:12:02.351 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T07:12:02.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:12:02.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:02.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:12:02.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:12:02.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:12:02.385 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:02.366+0000 7f5df3f03f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:02.385 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:02.370+0000 7f5df3f03f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:02.386 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:02.370+0000 7f5df3f03f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:02.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:12:02.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:12:02.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:12:02.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:12:02.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:12:02.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:02.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:12:02.699 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:12:02.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:02.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:02.936 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:02.922+0000 7f5df3f03f40 -1 Falling back to public interface 2021-07-26T07:12:03.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:04.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:04.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:04.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:12:04.073 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:12:04.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:04.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:04.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:05.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:05.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:05.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:05.428 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:12:05.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:05.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:05.583 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:05.570+0000 7f5df3f03f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:12:05.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:06.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:06.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:06.894 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:12:06.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:06.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:06.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:12:07.275 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/290701,v1:127.0.0.1:6819/290701] [v2:127.0.0.1:6820/290701,v1:127.0.0.1:6821/290701] exists,up 51919392-e5b8-42b8-b6d9-a6fbdeb6dcfd 2021-07-26T07:12:07.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:07.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:07.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:07.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:12:07.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2021-07-26T07:12:07.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:12:07.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:12:07.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T07:12:07.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:12:07.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2021-07-26T07:12:07.281 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:12:07.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:12:07.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:12:07.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:12:07.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-07-26T07:12:07.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-07-26T07:12:07.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:12:07.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:12:07.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:12:07.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:12:07.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:12:07.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:07.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:07.284 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:07.284 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:07.285 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:12:07.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:12:07.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:12:07.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:12:07.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:12:07.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:12:07.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:12:07.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:12:07.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:12:07.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:12:07.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:07.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:12:07.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:12:07.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2021-07-26T07:12:07.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:12:07.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=973b33f2-0cda-4aa6-83fa-84fd01db250a 2021-07-26T07:12:07.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 973b33f2-0cda-4aa6-83fa-84fd01db250a' 2021-07-26T07:12:07.289 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 973b33f2-0cda-4aa6-83fa-84fd01db250a 2021-07-26T07:12:07.290 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:12:07.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHYP5gIqAmERAAd2mvv1nqzmXCouIdfIDGqg== 2021-07-26T07:12:07.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHYP5gIqAmERAAd2mvv1nqzmXCouIdfIDGqg=="}' 2021-07-26T07:12:07.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 973b33f2-0cda-4aa6-83fa-84fd01db250a -i td/osd-recovery-stats/3/new.json 2021-07-26T07:12:07.846 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:12:07.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2021-07-26T07:12:07.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBHYP5gIqAmERAAd2mvv1nqzmXCouIdfIDGqg== --osd-uuid 973b33f2-0cda-4aa6-83fa-84fd01db250a 2021-07-26T07:12:07.894 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:07.882+0000 7facf9113f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:07.897 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:07.886+0000 7facf9113f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:07.899 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:07.886+0000 7facf9113f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:07.900 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:07.886+0000 7facf9113f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:12:07.900 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:07.886+0000 7facf9113f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:12:07.901 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:07.886+0000 7facf9113f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:12:08.009 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:07.994+0000 7facf9113f40 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-07-26T07:12:13.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-07-26T07:12:13.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:12:13.905 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T07:12:13.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T07:12:13.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:12:14.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T07:12:14.414 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T07:12:14.414 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:14.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:12:14.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:12:14.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:12:14.435 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:14.422+0000 7f409a5e7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:14.438 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:14.426+0000 7f409a5e7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:14.443 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:14.426+0000 7f409a5e7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:14.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T07:12:14.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:12:14.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:12:14.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:12:14.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:12:14.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:14.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:12:14.816 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:12:14.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:14.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:12:14.977 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:14.962+0000 7f409a5e7f40 -1 Falling back to public interface 2021-07-26T07:12:15.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:16.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:16.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:16.228 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:12:16.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:12:16.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:16.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:12:16.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:17.521 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:17.506+0000 7f409a5e7f40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:12:17.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:17.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:17.623 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:12:17.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:17.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:17.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:12:18.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:19.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:19.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:19.011 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:12:19.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:19.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:19.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:12:19.352 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/292107,v1:127.0.0.1:6827/292107] [v2:127.0.0.1:6828/292107,v1:127.0.0.1:6829/292107] exists,up 973b33f2-0cda-4aa6-83fa-84fd01db250a 2021-07-26T07:12:19.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:19.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:19.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:19.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:12:19.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2021-07-26T07:12:19.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:12:19.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:12:19.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T07:12:19.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:12:19.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2021-07-26T07:12:19.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:12:19.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:12:19.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:12:19.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:12:19.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-07-26T07:12:19.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-07-26T07:12:19.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:12:19.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:12:19.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:12:19.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:12:19.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:12:19.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:19.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:19.362 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:19.362 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:19.362 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:12:19.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:12:19.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:12:19.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:12:19.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:12:19.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:12:19.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:12:19.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:12:19.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:12:19.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:12:19.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:19.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:12:19.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:12:19.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2021-07-26T07:12:19.366 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:12:19.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7d640da-dfbf-436c-8893-b7f0494bf4dd 2021-07-26T07:12:19.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 d7d640da-dfbf-436c-8893-b7f0494bf4dd' 2021-07-26T07:12:19.367 INFO:tasks.workunit.client.0.smithi094.stdout:add osd4 d7d640da-dfbf-436c-8893-b7f0494bf4dd 2021-07-26T07:12:19.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:12:19.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTYP5gaVL/FRAAEkVGoP2+ZOgk0jFee6jFoQ== 2021-07-26T07:12:19.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTYP5gaVL/FRAAEkVGoP2+ZOgk0jFee6jFoQ=="}' 2021-07-26T07:12:19.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7d640da-dfbf-436c-8893-b7f0494bf4dd -i td/osd-recovery-stats/4/new.json 2021-07-26T07:12:19.989 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T07:12:20.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2021-07-26T07:12:20.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBTYP5gaVL/FRAAEkVGoP2+ZOgk0jFee6jFoQ== --osd-uuid d7d640da-dfbf-436c-8893-b7f0494bf4dd 2021-07-26T07:12:20.035 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:20.022+0000 7f38b7470f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:20.050 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:20.038+0000 7f38b7470f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:20.052 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:20.038+0000 7f38b7470f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:20.054 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:20.038+0000 7f38b7470f40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:12:20.054 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:20.038+0000 7f38b7470f40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:12:20.054 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:20.038+0000 7f38b7470f40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:12:20.095 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:20.082+0000 7f38b7470f40 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-07-26T07:12:26.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-07-26T07:12:26.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:12:26.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T07:12:26.313 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd4 key to auth repository 2021-07-26T07:12:26.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:12:26.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T07:12:26.814 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.4 2021-07-26T07:12:26.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:12:26.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:26.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:12:26.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:12:26.846 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:26.833+0000 7f1d4e7c8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:26.852 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:26.833+0000 7f1d4e7c8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:26.853 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:26.833+0000 7f1d4e7c8f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:27.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T07:12:27.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:12:27.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T07:12:27.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:12:27.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:12:27.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:27.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:12:27.138 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:12:27.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:27.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:12:27.392 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:27.377+0000 7f1d4e7c8f40 -1 Falling back to public interface 2021-07-26T07:12:27.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:28.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:28.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:28.537 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:12:28.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:12:28.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:28.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:12:28.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:29.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:29.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:29.899 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:12:29.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:29.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:29.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:12:30.049 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:30.033+0000 7f1d4e7c8f40 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T07:12:30.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:31.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:31.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:31.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:31.274 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:12:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:31.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:12:31.663 INFO:tasks.workunit.client.0.smithi094.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/293513,v1:127.0.0.1:6835/293513] [v2:127.0.0.1:6836/293513,v1:127.0.0.1:6837/293513] exists,up d7d640da-dfbf-436c-8893-b7f0494bf4dd 2021-07-26T07:12:31.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:31.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:31.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:31.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:12:31.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2021-07-26T07:12:31.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:12:31.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:12:31.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T07:12:31.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:12:31.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2021-07-26T07:12:31.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:12:31.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:12:31.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:12:31.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:12:31.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-07-26T07:12:31.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-07-26T07:12:31.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:12:31.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:12:31.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:12:31.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:12:31.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:12:31.671 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:12:31.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:12:31.672 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:12:31.672 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:12:31.672 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:12:31.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:12:31.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:12:31.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:12:31.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:12:31.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:12:31.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:12:31.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:12:31.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:12:31.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:12:31.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:31.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:12:31.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:12:31.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2021-07-26T07:12:31.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:12:31.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b05e307a-3010-4735-95a4-5e4323e8f341 2021-07-26T07:12:31.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 b05e307a-3010-4735-95a4-5e4323e8f341' 2021-07-26T07:12:31.678 INFO:tasks.workunit.client.0.smithi094.stdout:add osd5 b05e307a-3010-4735-95a4-5e4323e8f341 2021-07-26T07:12:31.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:12:31.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfYP5gxSJoKBAAjaQOCnuV1TokBDhKizvBZQ== 2021-07-26T07:12:31.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfYP5gxSJoKBAAjaQOCnuV1TokBDhKizvBZQ=="}' 2021-07-26T07:12:31.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b05e307a-3010-4735-95a4-5e4323e8f341 -i td/osd-recovery-stats/5/new.json 2021-07-26T07:12:32.211 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T07:12:32.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2021-07-26T07:12:32.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBfYP5gxSJoKBAAjaQOCnuV1TokBDhKizvBZQ== --osd-uuid b05e307a-3010-4735-95a4-5e4323e8f341 2021-07-26T07:12:32.258 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:32.245+0000 7f87e2337f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:32.261 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:32.249+0000 7f87e2337f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:32.263 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:32.249+0000 7f87e2337f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:32.264 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:32.249+0000 7f87e2337f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:12:32.264 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:32.249+0000 7f87e2337f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:12:32.264 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:32.249+0000 7f87e2337f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:12:32.339 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:32.325+0000 7f87e2337f40 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-07-26T07:12:38.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-07-26T07:12:38.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:12:38.533 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd5 key to auth repository 2021-07-26T07:12:38.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T07:12:38.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:12:39.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T07:12:39.026 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.5 2021-07-26T07:12:39.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:12:39.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:12:39.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:12:39.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:12:39.055 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:39.041+0000 7f5cfa2a5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:39.058 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:39.045+0000 7f5cfa2a5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:39.061 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:39.045+0000 7f5cfa2a5f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:12:39.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T07:12:39.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:12:39.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T07:12:39.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:12:39.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:12:39.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:39.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:12:39.352 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:12:39.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:39.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:12:39.592 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:39.577+0000 7f5cfa2a5f40 -1 Falling back to public interface 2021-07-26T07:12:39.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:40.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:40.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:40.754 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:12:40.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:12:40.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:40.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:12:41.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:42.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:42.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:42.116 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:12:42.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:12:42.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:42.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:12:42.190 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:12:42.177+0000 7f5cfa2a5f40 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T07:12:42.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:12:43.536 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:12:43.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:12:43.537 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:12:43.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:12:43.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:12:43.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:12:43.947 INFO:tasks.workunit.client.0.smithi094.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/294919,v1:127.0.0.1:6843/294919] [v2:127.0.0.1:6844/294919,v1:127.0.0.1:6845/294919] exists,up b05e307a-3010-4735-95a4-5e4323e8f341 2021-07-26T07:12:43.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:12:43.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:12:43.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:12:43.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2021-07-26T07:12:43.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T07:12:44.619 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T07:12:44.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:12:45.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2021-07-26T07:12:46.701 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 3 2021-07-26T07:12:46.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2021-07-26T07:12:47.882 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 min_size to 1 2021-07-26T07:12:47.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2021-07-26T07:12:47.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:12:47.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:12:47.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:12:47.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:12:47.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:12:47.898 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:12:47.898 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:12:47.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:12:47.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:12:47.899 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:12:48.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:12:48.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:12:48.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:12:48.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:12:48.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:12:48.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:12:48.510 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:12:48.511 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:12:48.511 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:12:48.511 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:12:48.512 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:12:48.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:12:48.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:48.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:12:48.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836496 2021-07-26T07:12:48.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836496 2021-07-26T07:12:48.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496' 2021-07-26T07:12:48.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:48.743 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:12:48.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672974 2021-07-26T07:12:48.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672974 2021-07-26T07:12:48.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-07-26T07:12:48.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:48.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:12:49.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509451 2021-07-26T07:12:49.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509451 2021-07-26T07:12:49.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509451' 2021-07-26T07:12:49.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:49.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:12:49.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345929 2021-07-26T07:12:49.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345929 2021-07-26T07:12:49.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509451 3-85899345929' 2021-07-26T07:12:49.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:49.188 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:12:49.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182406 2021-07-26T07:12:49.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182406 2021-07-26T07:12:49.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509451 3-85899345929 4-107374182406' 2021-07-26T07:12:49.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:12:49.334 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:12:49.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018884 2021-07-26T07:12:49.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018884 2021-07-26T07:12:49.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509451 3-85899345929 4-107374182406 5-128849018884' 2021-07-26T07:12:49.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:49.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836496 2021-07-26T07:12:49.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:49.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:12:49.483 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836496 2021-07-26T07:12:49.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:49.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836496 2021-07-26T07:12:49.485 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836496 2021-07-26T07:12:49.485 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-07-26T07:12:49.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:12:49.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836496 -lt 21474836496 2021-07-26T07:12:49.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:49.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672974 2021-07-26T07:12:49.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:49.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:12:49.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:49.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672974 2021-07-26T07:12:49.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672974 2021-07-26T07:12:49.849 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672974 2021-07-26T07:12:49.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-07-26T07:12:49.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:12:50.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672974 2021-07-26T07:12:50.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:50.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509451 2021-07-26T07:12:50.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:50.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:12:50.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509451 2021-07-26T07:12:50.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:50.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509451 2021-07-26T07:12:50.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-07-26T07:12:50.235 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509451 2021-07-26T07:12:50.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:12:50.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509451 -lt 64424509451 2021-07-26T07:12:50.630 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:50.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345929 2021-07-26T07:12:50.631 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:50.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:12:50.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345929 2021-07-26T07:12:50.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:50.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345929 2021-07-26T07:12:50.636 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2021-07-26T07:12:50.637 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345929 2021-07-26T07:12:50.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:12:51.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345929 -lt 85899345929 2021-07-26T07:12:51.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:51.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182406 2021-07-26T07:12:51.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:51.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:12:51.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182406 2021-07-26T07:12:51.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:51.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182406 2021-07-26T07:12:51.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-07-26T07:12:51.023 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182406 2021-07-26T07:12:51.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:12:51.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182406 2021-07-26T07:12:51.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:12:51.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018884 2021-07-26T07:12:51.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:12:51.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:12:51.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018884 2021-07-26T07:12:51.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:12:51.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018884 2021-07-26T07:12:51.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2021-07-26T07:12:51.403 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018884 2021-07-26T07:12:51.403 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:12:51.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018884 -lt 128849018884 2021-07-26T07:12:51.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:12:51.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:12:51.859 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:12:52.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:12:52.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:12:52.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:12:52.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:12:52.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:12:52.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:12:52.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:12:52.394 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:12:52.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:12:52.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:12:52.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:12:52.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:12:53.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:12:53.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:12:53.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:12:53.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2021-07-26T07:12:53.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2021-07-26T07:12:53.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:12:53.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:12:53.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:12:53.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:12:53.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2021-07-26T07:12:53.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2021-07-26T07:12:53.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test 2021-07-26T07:12:53.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T07:12:53.786 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test obj1 2021-07-26T07:12:53.786 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:12:53.787 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:12:53.787 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:12:53.787 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:12:54.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:12:54.178 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:12:54.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test obj1 2021-07-26T07:12:54.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2021-07-26T07:12:54.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2021-07-26T07:12:55.207 INFO:tasks.workunit.client.0.smithi094.stderr:noout is set 2021-07-26T07:12:55.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2021-07-26T07:12:56.206 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is set 2021-07-26T07:12:56.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2021-07-26T07:12:56.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 287889 2021-07-26T07:12:56.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2021-07-26T07:12:56.759 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T07:12:56.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2021-07-26T07:12:56.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2021-07-26T07:12:56.775 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2021-07-26T07:12:56.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:56.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2021-07-26T07:12:57.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:57.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2021-07-26T07:12:57.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:57.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2021-07-26T07:12:57.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:57.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2021-07-26T07:12:58.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2021-07-26T07:12:58.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2021-07-26T07:12:58.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2021-07-26T07:12:58.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2021-07-26T07:12:58.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2021-07-26T07:12:58.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2021-07-26T07:12:58.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2021-07-26T07:12:58.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:58.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2021-07-26T07:12:59.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2021-07-26T07:12:59.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2021-07-26T07:12:59.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2021-07-26T07:12:59.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2021-07-26T07:12:59.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2021-07-26T07:12:59.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2021-07-26T07:12:59.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2021-07-26T07:12:59.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:12:59.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2021-07-26T07:13:00.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:00.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2021-07-26T07:13:00.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:00.196 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2021-07-26T07:13:00.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:00.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2021-07-26T07:13:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:00.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2021-07-26T07:13:00.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:00.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2021-07-26T07:13:00.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:00.739 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2021-07-26T07:13:00.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:00.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2021-07-26T07:13:01.024 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2021-07-26T07:13:01.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2021-07-26T07:13:01.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2021-07-26T07:13:01.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2021-07-26T07:13:01.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2021-07-26T07:13:01.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2021-07-26T07:13:01.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2021-07-26T07:13:01.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:01.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2021-07-26T07:13:02.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2021-07-26T07:13:02.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2021-07-26T07:13:02.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2021-07-26T07:13:02.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2021-07-26T07:13:02.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2021-07-26T07:13:02.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2021-07-26T07:13:02.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2021-07-26T07:13:02.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:02.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2021-07-26T07:13:03.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2021-07-26T07:13:03.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2021-07-26T07:13:03.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2021-07-26T07:13:03.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2021-07-26T07:13:03.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2021-07-26T07:13:03.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2021-07-26T07:13:03.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2021-07-26T07:13:03.961 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:03.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2021-07-26T07:13:04.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2021-07-26T07:13:04.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2021-07-26T07:13:04.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2021-07-26T07:13:04.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2021-07-26T07:13:04.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2021-07-26T07:13:04.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2021-07-26T07:13:04.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2021-07-26T07:13:04.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:04.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2021-07-26T07:13:05.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2021-07-26T07:13:05.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2021-07-26T07:13:05.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2021-07-26T07:13:05.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2021-07-26T07:13:05.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2021-07-26T07:13:05.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2021-07-26T07:13:05.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2021-07-26T07:13:05.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:05.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2021-07-26T07:13:06.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2021-07-26T07:13:06.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2021-07-26T07:13:06.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.280 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2021-07-26T07:13:06.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2021-07-26T07:13:06.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2021-07-26T07:13:06.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2021-07-26T07:13:06.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2021-07-26T07:13:06.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2021-07-26T07:13:06.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:06.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2021-07-26T07:13:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:07.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2021-07-26T07:13:07.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:07.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2021-07-26T07:13:07.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:07.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2021-07-26T07:13:07.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:07.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2021-07-26T07:13:07.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:07.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2021-07-26T07:13:07.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:07.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2021-07-26T07:13:07.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:07.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2021-07-26T07:13:08.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2021-07-26T07:13:08.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2021-07-26T07:13:08.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2021-07-26T07:13:08.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2021-07-26T07:13:08.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2021-07-26T07:13:08.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2021-07-26T07:13:08.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2021-07-26T07:13:08.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2021-07-26T07:13:08.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:08.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2021-07-26T07:13:09.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:09.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2021-07-26T07:13:09.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:09.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2021-07-26T07:13:09.340 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:09.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2021-07-26T07:13:09.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:09.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2021-07-26T07:13:09.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:09.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2021-07-26T07:13:09.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:09.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2021-07-26T07:13:09.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-07-26T07:13:09.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2021-07-26T07:13:10.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2021-07-26T07:13:10.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 289296 2021-07-26T07:13:10.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2021-07-26T07:13:10.529 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already down. 2021-07-26T07:13:10.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2021-07-26T07:13:10.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-recovery-stats 2021-07-26T07:13:10.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:13:10.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=0 2021-07-26T07:13:10.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:13:10.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:13:10.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:13:10.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:13:10.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:13:10.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:13:10.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:13:10.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:13:10.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:13:10.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:13:10.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:13:10.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:13:10.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:13:10.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:13:10.543 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:13:10.544 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:13:10.544 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:10.544 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:13:10.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:13:10.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:13:10.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:13:10.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:13:10.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:13:10.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:13:10.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:13:10.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:13:10.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:13:10.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:13:10.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:13:10.548 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:13:10.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.0 2021-07-26T07:13:10.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:13:10.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-recovery-stats/0/whoami 2021-07-26T07:13:10.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 0 = 0 ']' 2021-07-26T07:13:10.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:13:10.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:13:10.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:13:10.567 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:10.552+0000 7f4e345a7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:10.586 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:10.572+0000 7f4e345a7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:10.587 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:10.572+0000 7f4e345a7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:10.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 0 2021-07-26T07:13:10.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:13:10.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:13:10.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:13:10.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:13:10.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:10.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:13:10.947 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:13:10.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:10.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:11.127 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:11.112+0000 7f4e345a7f40 -1 Falling back to public interface 2021-07-26T07:13:11.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:12.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:12.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:12.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:13:12.392 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:13:12.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:12.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:12.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:13.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:13.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:13.764 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:13:13.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:13:13.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:13.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:14.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:14.276 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:14.260+0000 7f4e345a7f40 -1 osd.0 42 log_to_monitors {default=true} 2021-07-26T07:13:15.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:15.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:15.140 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:13:15.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:13:15.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:15.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:13:15.625 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 44 last_clean_interval [5,42) [v2:127.0.0.1:6802/298281,v1:127.0.0.1:6803/298281] [v2:127.0.0.1:6804/298281,v1:127.0.0.1:6805/298281] exists,up e1334ce2-e601-44d5-be23-6b49bb85c5a7 2021-07-26T07:13:15.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:13:15.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:13:15.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:13:15.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2021-07-26T07:13:18.629 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2021-07-26T07:13:18.630 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2021-07-26T07:13:18.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:18.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2021-07-26T07:13:27.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:27.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2021-07-26T07:13:27.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:27.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2021-07-26T07:13:27.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:27.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2021-07-26T07:13:27.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:27.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2021-07-26T07:13:27.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:27.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2021-07-26T07:13:27.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:27.957 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2021-07-26T07:13:28.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2021-07-26T07:13:28.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2021-07-26T07:13:28.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2021-07-26T07:13:28.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2021-07-26T07:13:28.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2021-07-26T07:13:28.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2021-07-26T07:13:28.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2021-07-26T07:13:28.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:28.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2021-07-26T07:13:29.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2021-07-26T07:13:29.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2021-07-26T07:13:29.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2021-07-26T07:13:29.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2021-07-26T07:13:29.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2021-07-26T07:13:29.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2021-07-26T07:13:29.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2021-07-26T07:13:29.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2021-07-26T07:13:29.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:29.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2021-07-26T07:13:30.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2021-07-26T07:13:30.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.207 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2021-07-26T07:13:30.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2021-07-26T07:13:30.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2021-07-26T07:13:30.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2021-07-26T07:13:30.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2021-07-26T07:13:30.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2021-07-26T07:13:30.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:30.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2021-07-26T07:13:31.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2021-07-26T07:13:31.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2021-07-26T07:13:31.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2021-07-26T07:13:31.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2021-07-26T07:13:31.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2021-07-26T07:13:31.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2021-07-26T07:13:31.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2021-07-26T07:13:31.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:31.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2021-07-26T07:13:32.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2021-07-26T07:13:32.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2021-07-26T07:13:32.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2021-07-26T07:13:32.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2021-07-26T07:13:32.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2021-07-26T07:13:32.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2021-07-26T07:13:32.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2021-07-26T07:13:32.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2021-07-26T07:13:32.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:32.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2021-07-26T07:13:33.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:33.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2021-07-26T07:13:33.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:33.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2021-07-26T07:13:33.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:33.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2021-07-26T07:13:33.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:33.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2021-07-26T07:13:33.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:33.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2021-07-26T07:13:33.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:33.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2021-07-26T07:13:33.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:33.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2021-07-26T07:13:34.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2021-07-26T07:13:34.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2021-07-26T07:13:34.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2021-07-26T07:13:34.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2021-07-26T07:13:34.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2021-07-26T07:13:34.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2021-07-26T07:13:34.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2021-07-26T07:13:34.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2021-07-26T07:13:34.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:34.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2021-07-26T07:13:35.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:35.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2021-07-26T07:13:35.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:35.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2021-07-26T07:13:35.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:35.320 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2021-07-26T07:13:35.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:35.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2021-07-26T07:13:35.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:35.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2021-07-26T07:13:35.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:35.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2021-07-26T07:13:35.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:35.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2021-07-26T07:13:36.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2021-07-26T07:13:36.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2021-07-26T07:13:36.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2021-07-26T07:13:36.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2021-07-26T07:13:36.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2021-07-26T07:13:36.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2021-07-26T07:13:36.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2021-07-26T07:13:36.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2021-07-26T07:13:36.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:36.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2021-07-26T07:13:37.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2021-07-26T07:13:37.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2021-07-26T07:13:37.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2021-07-26T07:13:37.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2021-07-26T07:13:37.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2021-07-26T07:13:37.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2021-07-26T07:13:37.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2021-07-26T07:13:37.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:37.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2021-07-26T07:13:38.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2021-07-26T07:13:38.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2021-07-26T07:13:38.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2021-07-26T07:13:38.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2021-07-26T07:13:38.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2021-07-26T07:13:38.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2021-07-26T07:13:38.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2021-07-26T07:13:38.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:38.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2021-07-26T07:13:39.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:39.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2021-07-26T07:13:39.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:39.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2021-07-26T07:13:39.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-07-26T07:13:39.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2021-07-26T07:13:39.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2021-07-26T07:13:39.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T07:13:39.380 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:13:39.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test obj1 2021-07-26T07:13:39.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:13:39.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2021-07-26T07:13:39.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2021-07-26T07:13:39.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test 2021-07-26T07:13:39.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj200 2021-07-26T07:13:39.793 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test obj200 2021-07-26T07:13:39.794 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:13:39.794 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj200 2021-07-26T07:13:39.794 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj200 2021-07-26T07:13:39.794 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:13:40.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=0 2021-07-26T07:13:40.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2021-07-26T07:13:40.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test obj200 2021-07-26T07:13:40.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2021-07-26T07:13:40.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2021-07-26T07:13:41.474 INFO:tasks.workunit.client.0.smithi094.stderr:noout is unset 2021-07-26T07:13:41.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2021-07-26T07:13:42.043 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already out. osd.2 is already out. 2021-07-26T07:13:42.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2021-07-26T07:13:42.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local dir=td/osd-recovery-stats 2021-07-26T07:13:42.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: shift 2021-07-26T07:13:42.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local id=1 2021-07-26T07:13:42.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: shift 2021-07-26T07:13:42.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:13:42.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:13:42.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:13:42.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:13:42.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:13:42.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:13:42.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:13:42.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:13:42.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --chdir=' 2021-07-26T07:13:42.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2021-07-26T07:13:42.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:13:42.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: get_asok_path 2021-07-26T07:13:42.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:13:42.062 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:13:42.064 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:13:42.064 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:13:42.065 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:13:42.065 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:13:42.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:13:42.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:13:42.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:13:42.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:13:42.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:13:42.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:13:42.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:13:42.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=' ' 2021-07-26T07:13:42.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph_args+= 2021-07-26T07:13:42.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:13:42.068 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:13:42.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: echo start osd.1 2021-07-26T07:13:42.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:13:42.069 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: cat td/osd-recovery-stats/1/whoami 2021-07-26T07:13:42.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: '[' 1 = 1 ']' 2021-07-26T07:13:42.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: ceph osd dump --format=json 2021-07-26T07:13:42.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: jq '.flags_set[]' 2021-07-26T07:13:42.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: grep -q '"noup"' 2021-07-26T07:13:42.089 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:42.072+0000 7f914cdaef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:42.092 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:42.076+0000 7f914cdaef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:42.098 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:42.076+0000 7f914cdaef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:13:42.380 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:42.364+0000 7f914cdaef40 -1 Falling back to public interface 2021-07-26T07:13:42.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:883: activate_osd: wait_for_osd up 1 2021-07-26T07:13:42.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:13:42.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:13:42.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:13:42.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:13:42.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:42.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:13:42.446 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:13:42.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:42.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:13:42.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:43.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:43.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:43.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:13:43.902 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:13:43.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:43.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:13:44.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:45.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:45.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:45.272 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:13:45.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:13:45.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:45.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:13:45.357 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:13:45.344+0000 7f914cdaef40 -1 osd.1 45 log_to_monitors {default=true} 2021-07-26T07:13:45.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:13:46.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:13:46.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:13:46.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:13:46.719 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:13:46.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:13:46.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:13:47.082 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up out weight 0 up_from 58 up_thru 44 down_at 46 last_clean_interval [10,45) [v2:127.0.0.1:6810/300988,v1:127.0.0.1:6811/300988] [v2:127.0.0.1:6812/300988,v1:127.0.0.1:6813/300988] exists,up b9e4eea4-9ce1-442c-bd05-1362418efb04 2021-07-26T07:13:47.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:13:47.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:13:47.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:13:47.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2021-07-26T07:13:50.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2021-07-26T07:13:50.969 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 4 2021-07-26T07:13:50.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2021-07-26T07:13:52.570 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is unset 2021-07-26T07:13:52.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2021-07-26T07:13:52.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:13:52.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:13:52.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:13:52.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:13:52.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2021-07-26T07:13:53.086 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T07:13:53.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2021-07-26T07:13:55.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2021-07-26T07:13:55.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:13:55.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:13:55.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:13:55.098 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:13:55.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:13:55.099 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:13:55.099 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:13:55.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:13:55.100 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:13:55.101 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:13:55.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:13:55.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:13:55.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:13:55.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:13:55.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:13:55.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:13:55.704 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:13:55.704 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:13:55.704 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:13:55.705 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:13:55.705 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:13:55.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:13:55.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:55.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:13:55.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=210453397515 2021-07-26T07:13:55.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 210453397515 2021-07-26T07:13:55.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-210453397515' 2021-07-26T07:13:55.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:55.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:13:55.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=249108103173 2021-07-26T07:13:55.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 249108103173 2021-07-26T07:13:55.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-210453397515 1-249108103173' 2021-07-26T07:13:55.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:55.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:13:56.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509466 2021-07-26T07:13:56.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509466 2021-07-26T07:13:56.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-210453397515 1-249108103173 2-64424509466' 2021-07-26T07:13:56.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:56.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:13:56.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345944 2021-07-26T07:13:56.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345944 2021-07-26T07:13:56.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-210453397515 1-249108103173 2-64424509466 3-85899345944' 2021-07-26T07:13:56.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:56.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:13:56.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182421 2021-07-26T07:13:56.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182421 2021-07-26T07:13:56.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-210453397515 1-249108103173 2-64424509466 3-85899345944 4-107374182421' 2021-07-26T07:13:56.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:13:56.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:13:56.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018899 2021-07-26T07:13:56.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018899 2021-07-26T07:13:56.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-210453397515 1-249108103173 2-64424509466 3-85899345944 4-107374182421 5-128849018899' 2021-07-26T07:13:56.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:56.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-210453397515 2021-07-26T07:13:56.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:56.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:13:56.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:56.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-210453397515 2021-07-26T07:13:56.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=210453397515 2021-07-26T07:13:56.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 210453397515' 2021-07-26T07:13:56.578 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 210453397515 2021-07-26T07:13:56.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:56.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 210453397513 -lt 210453397515 2021-07-26T07:13:56.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:13:57.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:13:57.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:13:58.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 210453397515 -lt 210453397515 2021-07-26T07:13:58.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:58.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-249108103173 2021-07-26T07:13:58.295 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:58.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:13:58.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-249108103173 2021-07-26T07:13:58.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:58.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=249108103173 2021-07-26T07:13:58.300 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 249108103173 2021-07-26T07:13:58.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2021-07-26T07:13:58.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:13:58.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 249108103173 -lt 249108103173 2021-07-26T07:13:58.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:58.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509466 2021-07-26T07:13:58.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:58.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:13:58.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509466 2021-07-26T07:13:58.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:58.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509466 2021-07-26T07:13:58.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2021-07-26T07:13:58.725 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509466 2021-07-26T07:13:58.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:13:59.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509467 -lt 64424509466 2021-07-26T07:13:59.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:59.190 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345944 2021-07-26T07:13:59.190 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:59.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:13:59.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345944 2021-07-26T07:13:59.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:59.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345944 2021-07-26T07:13:59.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345944' 2021-07-26T07:13:59.196 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345944 2021-07-26T07:13:59.196 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:13:59.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345944 -lt 85899345944 2021-07-26T07:13:59.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:59.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182421 2021-07-26T07:13:59.555 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:59.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:13:59.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182421 2021-07-26T07:13:59.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:59.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182421 2021-07-26T07:13:59.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2021-07-26T07:13:59.561 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182421 2021-07-26T07:13:59.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:13:59.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182422 -lt 107374182421 2021-07-26T07:13:59.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:13:59.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:13:59.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018899 2021-07-26T07:13:59.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:13:59.992 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018899 2021-07-26T07:13:59.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:13:59.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018899 2021-07-26T07:13:59.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018899' 2021-07-26T07:13:59.996 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018899 2021-07-26T07:13:59.996 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:14:00.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018900 -lt 128849018899 2021-07-26T07:14:00.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:14:00.381 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:00.382 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:00.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:14:00.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:00.946 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:00.946 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:00.946 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:00.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:00.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:00.947 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:01.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:01.342 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:01.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:01.343 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:01.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:01.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:14:01.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:14:01.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:14:01.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:01.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:01.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:01.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:01.912 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:01.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:01.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:01.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:01.913 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:01.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:02.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:02.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:02.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:02.275 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:02.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:02.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:02.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:02.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:02.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:02.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:02.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:02.778 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:02.779 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:03.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:14:03.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:14:03.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:03.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:03.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:03.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:03.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:03.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:03.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:03.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:03.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:03.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:03.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:03.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:03.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:03.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:03.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:04.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:04.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:04.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:04.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:04.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:04.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:04.193 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:04.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:04.194 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:04.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:14:04.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:14:04.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:04.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:04.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:04.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:04.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:04.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:04.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:04.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:04.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:04.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:04.811 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:05.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:05.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:05.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:05.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:05.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:05.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:05.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:05.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:05.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:05.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:05.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:05.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:05.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:06.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:06.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:06.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:06.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:06.195 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:06.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:06.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:06.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:06.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:06.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:06.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:06.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:06.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:06.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:06.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:06.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:06.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:07.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:07.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:07.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:07.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:07.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:07.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:07.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:07.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:07.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:07.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:07.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:07.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:07.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:07.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:07.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:07.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:07.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:07.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:07.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:07.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:07.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:07.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:08.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:08.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:08.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:08.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:08.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:08.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:08.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:08.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:08.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:08.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:08.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:08.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:08.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:09.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:09.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:09.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:09.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:09.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:09.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:09.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:09.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:09.271 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:09.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:09.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:09.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:09.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:09.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:09.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:09.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:09.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:10.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:10.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:10.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:10.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:10.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:10.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:10.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:10.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:10.215 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:10.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:14:10.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:14:10.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:10.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:10.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:10.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:10.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:10.889 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:10.890 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:10.890 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:10.890 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:10.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:10.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:11.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:11.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:11.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:11.255 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:11.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:11.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:11.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:11.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:11.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:11.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:11.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:11.835 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:11.835 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:12.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:12.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:12.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:12.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:12.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:12.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:12.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:12.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:12.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:12.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:12.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:12.497 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:12.498 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:12.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:12.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:12.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:12.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:13.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:13.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:13.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:13.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:13.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:13.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:13.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:13.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:13.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:13.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:13.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:13.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:13.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:13.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:14.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:14.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:14.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:14.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:14.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:14.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:14.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:14.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:14.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:14.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:14.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:14.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:14.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:14.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:14.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:14.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:14.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:14.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:14.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:14.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:14.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:15.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:15.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:15.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:15.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:15.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:15.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:15.513 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:15.513 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:15.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:15.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:15.514 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:15.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:15.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:15.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:15.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:15.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:15.962 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:16.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:16.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:16.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:16.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:16.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:16.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:16.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:16.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:16.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:16.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:16.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:17.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:17.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:17.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:17.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:17.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:17.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:17.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:17.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:17.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:17.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:17.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:17.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:17.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:17.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:17.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:17.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:17.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:17.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:17.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:17.888 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:17.888 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:18.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:18.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:18.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:18.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:18.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:18.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:18.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:18.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:18.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:18.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:18.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:18.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:18.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:18.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:18.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:18.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:18.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:19.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:19.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:19.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:19.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:19.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:19.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:19.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:19.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:19.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:19.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:19.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:19.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:19.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:19.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:20.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:20.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:20.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:20.049 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:20.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:20.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:20.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:20.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:20.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:20.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:20.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:20.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:20.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:20.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:20.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:20.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:20.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:20.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:20.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:21.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:21.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:21.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:21.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:21.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:21.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:21.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:21.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:21.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:21.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:21.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:21.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:21.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:22.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:22.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:22.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:22.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:22.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:22.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:22.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:22.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:22.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:22.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:22.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:22.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:22.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:23.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:23.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:23.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:23.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:23.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:23.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:23.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:23.218 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:23.218 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:23.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:23.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:23.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:23.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:23.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:23.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:23.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:23.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:24.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:24.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:24.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:24.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:24.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:24.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:24.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:24.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:24.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:24.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:14:24.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:14:24.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:24.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:24.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:24.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:24.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:24.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:24.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:24.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:24.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:24.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:24.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:25.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:25.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:25.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:25.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:25.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:25.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:25.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:25.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:25.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:25.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:25.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:25.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:25.506 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:25.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:25.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:25.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:25.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:25.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:26.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:26.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:26.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:26.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:26.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:26.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:26.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:26.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:26.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:26.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:26.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:26.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:26.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:26.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:26.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:26.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:26.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:26.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:26.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:26.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:26.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:27.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:27.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:27.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:27.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:27.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:27.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:27.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:27.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:27.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:27.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:27.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:27.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:27.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:27.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:27.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:27.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:27.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:28.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:28.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:28.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:28.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:28.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:28.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:28.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:28.291 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:28.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:28.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:28.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:28.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:28.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:28.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:28.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:28.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:28.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:28.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:28.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:28.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:28.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:28.916 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:29.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:29.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:29.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:29.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:29.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:29.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:29.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:29.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:29.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:29.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:29.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:29.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:29.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:30.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:30.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:30.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:30.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:30.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:30.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:30.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:30.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:30.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:30.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:30.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:30.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:30.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:30.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:30.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:30.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:30.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:31.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:31.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:31.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:31.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:31.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:31.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:31.093 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:31.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:31.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:31.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:14:31.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:14:31.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:31.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:31.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:31.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:31.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:31.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:31.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:31.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:31.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:31.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:31.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:32.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:32.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:32.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:32.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:32.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:32.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:32.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:32.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:32.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:32.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:32.560 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:32.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:32.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:14:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:14:33.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:33.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:33.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:33.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:33.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:33.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:33.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:33.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:33.235 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:33.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:33.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:33.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:33.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:33.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:33.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:34.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:34.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:34.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:34.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:34.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:34.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:34.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:34.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:34.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:34.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:34.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:34.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:34.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:34.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:34.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:34.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:34.709 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:34.709 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:34.710 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:34.710 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:34.711 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:34.711 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:35.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:35.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:35.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:35.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:35.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:35.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:35.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:35.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:35.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:35.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:35.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:35.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:35.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:36.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=6 2021-07-26T07:14:36.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 6 '!=' null 2021-07-26T07:14:36.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:36.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:36.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:36.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:36.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:36.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:36.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:36.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:36.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:36.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:36.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:36.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:36.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:36.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:36.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:37.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:37.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:37.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:37.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:37.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:37.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:37.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:37.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:37.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:38.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:38.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:38.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:38.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:38.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:38.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:38.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:38.116 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:38.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:38.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:38.117 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:38.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:38.118 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:38.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:38.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:38.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:38.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:38.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:38.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:38.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:38.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:38.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:38.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:38.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:38.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:38.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:39.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:39.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:39.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:39.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:39.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:39.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:39.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:39.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:39.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:39.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:39.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:39.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:39.738 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:40.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:40.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:40.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:40.096 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:40.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:40.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:40.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:40.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:40.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:40.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:40.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:40.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:40.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:41.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:14:41.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:14:41.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:41.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:41.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:41.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:41.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:41.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:41.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:41.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:41.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:41.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:41.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:41.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:41.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:41.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:41.604 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:42.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:42.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:42.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:42.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:42.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:42.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:42.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:42.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:42.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:42.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=6 2021-07-26T07:14:42.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 6 '!=' null 2021-07-26T07:14:42.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:42.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:42.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:42.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:42.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:42.735 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:42.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:42.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:42.736 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:42.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:42.737 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:43.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:43.070 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:43.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:43.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:43.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:43.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:43.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:43.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:43.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:43.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:43.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:14:43.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:14:43.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:43.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:43.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:44.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:44.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:44.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:44.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:44.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:44.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:44.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:44.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:44.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:14:44.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:44.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:44.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:44.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:14:44.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:14:44.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:14:44.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:14:44.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:14:44.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:14:44.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:14:44.891 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:14:44.892 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:14:45.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:14:45.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:14:45.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:14:45.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:14:45.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:14:45.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:14:45.631 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:14:45.632 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:14:45.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:14:45.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:14:45.633 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:14:45.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:14:45.634 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:14:45.987 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:14:45.988 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:14:45.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:14:45.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:14:46.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:14:46.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:14:46.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:14:46.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2021-07-26T07:14:46.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:14:46.447 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:14:46.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:14:46.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:14:46.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2021-07-26T07:14:46.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2021-07-26T07:14:46.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-07-26T07:14:46.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-07-26T07:14:46.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2021-07-26T07:14:46.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-07-26T07:14:46.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2021-07-26T07:14:46.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-07-26T07:14:46.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2021-07-26T07:14:46.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-07-26T07:14:46.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2021-07-26T07:14:46.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2021-07-26T07:14:46.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-07-26T07:14:46.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-07-26T07:14:46.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-07-26T07:14:46.809 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.809 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-07-26T07:14:46.809 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-07-26T07:14:46.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2021-07-26T07:14:46.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-07-26T07:14:46.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:14:46.843 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.844 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2021-07-26T07:14:46.844 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-07-26T07:14:46.844 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-07-26T07:14:46.845 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:14:46.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2021-07-26T07:14:46.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2021-07-26T07:14:46.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2021-07-26T07:14:46.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:14:46.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2021-07-26T07:14:46.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:14:46.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:14:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-07-26T07:14:46.852 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:14:46.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-07-26T07:14:46.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-07-26T07:14:46.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:14:46.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:14:46.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:14:46.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:14:46.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-07-26T07:14:46.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:14:46.874 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.874 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2021-07-26T07:14:46.874 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-07-26T07:14:46.875 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-07-26T07:14:46.875 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:14:46.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2021-07-26T07:14:46.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2021-07-26T07:14:46.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2021-07-26T07:14:46.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:14:46.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2021-07-26T07:14:46.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:14:46.881 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:14:46.882 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.882 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-07-26T07:14:46.882 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:14:46.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-07-26T07:14:46.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-07-26T07:14:46.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:14:46.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:14:46.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:14:46.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:14:46.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2021-07-26T07:14:46.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2021-07-26T07:14:46.901 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:14:46.901 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.901 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2021-07-26T07:14:46.902 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2021-07-26T07:14:46.902 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:14:46.902 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2021-07-26T07:14:46.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2021-07-26T07:14:46.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2021-07-26T07:14:46.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2021-07-26T07:14:46.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:14:46.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2021-07-26T07:14:46.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:14:46.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:14:46.910 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2021-07-26T07:14:46.910 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:14:46.910 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2021-07-26T07:14:46.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2021-07-26T07:14:46.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2021-07-26T07:14:46.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:14:46.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:14:46.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:14:46.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:14:46.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2021-07-26T07:14:46.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T07:14:46.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T07:14:47.410 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T07:14:47.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2021-07-26T07:14:47.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:14:47.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:14:47.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:14:47.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:14:47.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:14:47.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:14:47.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-07-26T07:14:47.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:14:47.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:14:47.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:14:47.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:14:47.746 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:14:47.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:14:47.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:14:47.747 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:14:47.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:14:47.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:14:47.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:14:47.763 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:14:47.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:14:47.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:14:47.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:14:47.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:14:47.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:14:47.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:14:47.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:14:47.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:14:47.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:14:47.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:14:47.791 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:14:47.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:47.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:47.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:14:47.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:14:47.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:14:47.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-07-26T07:14:47.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-07-26T07:14:47.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-07-26T07:14:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-07-26T07:14:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:14:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:14:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:14:47.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:14:47.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:14:47.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:14:47.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:14:47.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:14:47.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:14:47.798 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:14:47.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:14:47.799 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:14:47.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:14:47.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:14:47.802 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:14:47.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:14:47.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:14:47.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:14:47.804 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:14:47.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:14:47.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:14:47.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:14:47.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:14:47.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:47.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:47.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:14:47.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:14:47.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:14:47.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-07-26T07:14:47.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:14:47.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:47.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:47.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257957 2021-07-26T07:14:47.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:14:47.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:14:47.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:14:47.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-07-26T07:14:47.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2021-07-26T07:14:47.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2021-07-26T07:14:47.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2021-07-26T07:14:47.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2021-07-26T07:14:47.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2021-07-26T07:14:47.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2021-07-26T07:14:47.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2021-07-26T07:14:47.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-07-26T07:14:47.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:14:47.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:14:47.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:14:47.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-07-26T07:14:47.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-07-26T07:14:48.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:14:48.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:14:48.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:14:48.232 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:14:48.232 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:48.232 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:48.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:14:48.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:14:48.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:14:48.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:14:48.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:14:48.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:14:48.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:14:48.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:14:48.531 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:14:48.531 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:14:48.531 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:14:48.532 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:14:48.533 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:48.533 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:48.533 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:14:48.533 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:14:48.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get fsid 2021-07-26T07:14:48.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:14:48.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:14:48.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:14:48.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:14:48.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:14:48.626 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:14:48.626 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:14:48.627 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:14:48.627 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:14:48.627 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:48.627 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:48.627 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:14:48.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:14:48.628 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get mon_host 2021-07-26T07:14:48.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2021-07-26T07:14:48.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2021-07-26T07:14:48.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:14:48.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:14:48.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:14:48.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2021-07-26T07:14:48.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:14:49.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:14:49.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:14:49.605 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:14:49.606 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:14:49.606 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:49.607 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:49.607 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:14:49.608 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:14:49.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:14:49.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2021-07-26T07:14:49.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:14:49.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:14:49.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:14:49.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:14:49.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:14:49.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:14:49.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:14:49.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:14:49.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:14:49.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:14:49.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:14:49.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:14:49.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:14:49.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:14:49.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:14:49.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:14:49.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:14:49.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:14:49.650 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:14:49.651 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:14:49.651 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:14:49.651 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:14:49.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:14:49.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:14:49.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:14:49.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:14:49.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:14:49.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:14:49.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:14:49.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:14:49.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:14:49.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:14:49.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:14:49.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:14:49.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:14:49.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e0fc55f1-2347-4e04-abc9-b943ba3175f3 2021-07-26T07:14:49.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e0fc55f1-2347-4e04-abc9-b943ba3175f3' 2021-07-26T07:14:49.655 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 e0fc55f1-2347-4e04-abc9-b943ba3175f3 2021-07-26T07:14:49.655 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:14:49.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpYP5gc7AmJxAA8WRGg4swKAdZkan+ezFn6w== 2021-07-26T07:14:49.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpYP5gc7AmJxAA8WRGg4swKAdZkan+ezFn6w=="}' 2021-07-26T07:14:49.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e0fc55f1-2347-4e04-abc9-b943ba3175f3 -i td/osd-recovery-stats/0/new.json 2021-07-26T07:14:49.997 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:14:50.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2021-07-26T07:14:50.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpYP5gc7AmJxAA8WRGg4swKAdZkan+ezFn6w== --osd-uuid e0fc55f1-2347-4e04-abc9-b943ba3175f3 2021-07-26T07:14:50.041 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:50.026+0000 7fc266ce2f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:50.060 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:50.046+0000 7fc266ce2f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:50.061 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:50.046+0000 7fc266ce2f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:50.062 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:50.046+0000 7fc266ce2f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:14:50.063 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:50.046+0000 7fc266ce2f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:14:50.063 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:50.046+0000 7fc266ce2f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:14:50.106 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:50.090+0000 7fc266ce2f40 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-07-26T07:14:55.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-07-26T07:14:55.836 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:14:55.837 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:14:55.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:14:55.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:14:56.341 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:14:56.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:14:56.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:14:56.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:14:56.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:14:56.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:14:56.369 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:56.354+0000 7f298e665f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:56.396 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:56.382+0000 7f298e665f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:56.397 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:56.382+0000 7f298e665f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:14:56.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:14:56.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:14:56.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:14:56.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:14:56.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:14:56.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:56.734 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:14:56.734 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:14:56.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:56.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:56.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:56.922+0000 7f298e665f40 -1 Falling back to public interface 2021-07-26T07:14:57.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:58.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:58.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:58.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:14:58.131 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:14:58.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:58.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:58.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:14:59.476 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:14:59.462+0000 7f298e665f40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:14:59.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:14:59.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:14:59.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:14:59.499 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:14:59.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:14:59.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:14:59.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:00.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:00.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:00.991 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:15:00.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:15:00.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:00.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:15:01.344 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/305813,v1:127.0.0.1:6803/305813] [v2:127.0.0.1:6804/305813,v1:127.0.0.1:6805/305813] exists,up e0fc55f1-2347-4e04-abc9-b943ba3175f3 2021-07-26T07:15:01.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:01.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:01.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:01.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2021-07-26T07:15:01.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:15:01.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:01.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:15:01.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:01.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:15:01.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:15:01.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:01.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:01.358 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:01.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:15:01.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:15:01.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:01.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:01.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:01.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:15:01.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:01.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:01.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:01.363 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:01.363 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:01.363 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:15:01.364 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:01.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:01.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:01.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:01.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:01.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:15:01.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:15:01.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:01.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:01.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:01.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:01.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:15:01.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:15:01.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:01.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4be5a16a-970e-4b99-965d-77f3558bc95b 2021-07-26T07:15:01.371 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 4be5a16a-970e-4b99-965d-77f3558bc95b 2021-07-26T07:15:01.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4be5a16a-970e-4b99-965d-77f3558bc95b' 2021-07-26T07:15:01.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:01.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1YP5g/ViXFRAA+6rhq5IKsSolCQEOPPjK0w== 2021-07-26T07:15:01.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1YP5g/ViXFRAA+6rhq5IKsSolCQEOPPjK0w=="}' 2021-07-26T07:15:01.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4be5a16a-970e-4b99-965d-77f3558bc95b -i td/osd-recovery-stats/1/new.json 2021-07-26T07:15:01.910 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:15:01.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2021-07-26T07:15:01.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1YP5g/ViXFRAA+6rhq5IKsSolCQEOPPjK0w== --osd-uuid 4be5a16a-970e-4b99-965d-77f3558bc95b 2021-07-26T07:15:01.953 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:01.938+0000 7ff8305aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:01.955 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:01.942+0000 7ff8305aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:01.957 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:01.942+0000 7ff8305aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:01.958 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:01.942+0000 7ff8305aaf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:15:01.958 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:01.942+0000 7ff8305aaf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:15:01.959 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:01.942+0000 7ff8305aaf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:15:02.032 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:02.018+0000 7ff8305aaf40 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-07-26T07:15:08.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-07-26T07:15:08.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:08.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:15:08.130 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:15:08.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:08.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:15:08.643 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:15:08.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:08.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:08.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:08.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:08.671 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:08.658+0000 7fd09ed22f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:08.692 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:08.678+0000 7fd09ed22f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:08.693 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:08.678+0000 7fd09ed22f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:09.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:15:09.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:09.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:15:09.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:09.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:09.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:09.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:09.019 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:15:09.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:09.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:09.237 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:09.222+0000 7fd09ed22f40 -1 Falling back to public interface 2021-07-26T07:15:09.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:10.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:10.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:10.380 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:15:10.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:10.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:10.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:10.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:11.783 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:11.769+0000 7fd09ed22f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:15:11.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:11.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:11.845 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:15:11.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:11.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:11.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:12.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:13.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:13.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:13.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:15:13.217 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:15:13.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:13.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:15:13.574 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/307222,v1:127.0.0.1:6811/307222] [v2:127.0.0.1:6812/307222,v1:127.0.0.1:6813/307222] exists,up 4be5a16a-970e-4b99-965d-77f3558bc95b 2021-07-26T07:15:13.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:13.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:13.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:13.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2021-07-26T07:15:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:15:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:15:13.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:13.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2021-07-26T07:15:13.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:15:13.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:13.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:13.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:13.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-07-26T07:15:13.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-07-26T07:15:13.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:13.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:13.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:13.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:15:13.582 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:13.582 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:13.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:13.583 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:13.583 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:13.583 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:15:13.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:13.585 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:13.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:13.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:13.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:13.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:15:13.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:15:13.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:13.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:13.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:13.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:13.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:15:13.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2021-07-26T07:15:13.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:13.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d028c451-b3a9-4ac4-a75d-b9257bab33a2 2021-07-26T07:15:13.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d028c451-b3a9-4ac4-a75d-b9257bab33a2' 2021-07-26T07:15:13.589 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 d028c451-b3a9-4ac4-a75d-b9257bab33a2 2021-07-26T07:15:13.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:13.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABYf5gqh04IxAAvDPA/HaSfhXQcWoty5m/xg== 2021-07-26T07:15:13.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABYf5gqh04IxAAvDPA/HaSfhXQcWoty5m/xg=="}' 2021-07-26T07:15:13.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d028c451-b3a9-4ac4-a75d-b9257bab33a2 -i td/osd-recovery-stats/2/new.json 2021-07-26T07:15:14.138 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:15:14.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2021-07-26T07:15:14.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQABYf5gqh04IxAAvDPA/HaSfhXQcWoty5m/xg== --osd-uuid d028c451-b3a9-4ac4-a75d-b9257bab33a2 2021-07-26T07:15:14.184 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:14.169+0000 7fcebfb9af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:14.186 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:14.173+0000 7fcebfb9af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:14.188 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:14.173+0000 7fcebfb9af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:14.188 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:14.173+0000 7fcebfb9af40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:15:14.189 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:14.173+0000 7fcebfb9af40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:15:14.189 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:14.173+0000 7fcebfb9af40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:15:14.268 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:14.253+0000 7fcebfb9af40 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-07-26T07:15:20.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-07-26T07:15:20.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:20.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:15:20.387 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T07:15:20.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:20.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:15:20.907 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T07:15:20.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:20.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:20.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:20.909 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:20.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:20.921+0000 7fe494e9ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:20.972 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:20.957+0000 7fe494e9ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:20.973 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:20.957+0000 7fe494e9ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:21.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:15:21.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:21.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:15:21.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:21.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:21.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:21.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:21.241 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:15:21.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:21.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:21.508 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:21.493+0000 7fe494e9ef40 -1 Falling back to public interface 2021-07-26T07:15:21.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:22.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:22.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:22.626 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:22.627 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:15:22.627 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:22.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:22.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:23.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:23.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:23.994 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:15:23.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:23.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:23.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:24.136 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:24.121+0000 7fe494e9ef40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:15:24.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:25.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:25.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:25.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:15:25.353 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:15:25.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:25.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:15:25.714 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/308629,v1:127.0.0.1:6819/308629] [v2:127.0.0.1:6820/308629,v1:127.0.0.1:6821/308629] exists,up d028c451-b3a9-4ac4-a75d-b9257bab33a2 2021-07-26T07:15:25.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:25.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:25.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:25.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2021-07-26T07:15:25.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:15:25.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:25.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T07:15:25.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:25.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2021-07-26T07:15:25.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:15:25.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:25.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:25.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:25.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-07-26T07:15:25.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-07-26T07:15:25.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:25.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:25.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:25.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:15:25.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:25.721 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:25.722 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:25.722 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:25.722 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:25.722 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:15:25.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:25.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:25.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:25.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:25.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:25.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:15:25.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:15:25.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:25.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:25.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:25.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:25.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:15:25.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2021-07-26T07:15:25.728 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:25.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a425b01d-eb5f-4429-85bd-8d8231ddc327 2021-07-26T07:15:25.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a425b01d-eb5f-4429-85bd-8d8231ddc327' 2021-07-26T07:15:25.729 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 a425b01d-eb5f-4429-85bd-8d8231ddc327 2021-07-26T07:15:25.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:25.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANYf5gpuWZKxAAd65dX/sm/B1a1U1H57aIJw== 2021-07-26T07:15:25.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANYf5gpuWZKxAAd65dX/sm/B1a1U1H57aIJw=="}' 2021-07-26T07:15:25.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a425b01d-eb5f-4429-85bd-8d8231ddc327 -i td/osd-recovery-stats/3/new.json 2021-07-26T07:15:26.245 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:15:26.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2021-07-26T07:15:26.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQANYf5gpuWZKxAAd65dX/sm/B1a1U1H57aIJw== --osd-uuid a425b01d-eb5f-4429-85bd-8d8231ddc327 2021-07-26T07:15:26.294 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:26.277+0000 7f05db364f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:26.298 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:26.281+0000 7f05db364f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:26.300 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:26.285+0000 7f05db364f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:26.301 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:26.285+0000 7f05db364f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:15:26.302 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:26.285+0000 7f05db364f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:15:26.302 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:26.285+0000 7f05db364f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:15:26.414 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:26.397+0000 7f05db364f40 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-07-26T07:15:32.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-07-26T07:15:32.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:32.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T07:15:32.738 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T07:15:32.738 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:33.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T07:15:33.211 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T07:15:33.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:33.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:33.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:33.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:33.242 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:33.225+0000 7f9ad4cf4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:33.244 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:33.229+0000 7f9ad4cf4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:33.245 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:33.229+0000 7f9ad4cf4f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:33.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T07:15:33.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:33.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:15:33.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:33.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:33.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:33.534 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:15:33.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:33.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:33.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:15:33.548 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:33.533+0000 7f9ad4cf4f40 -1 Falling back to public interface 2021-07-26T07:15:33.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:34.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:34.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:34.883 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:15:34.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:34.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:34.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:15:35.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:36.208 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:36.193+0000 7f9ad4cf4f40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:15:36.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:36.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:36.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:36.245 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:15:36.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:36.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:15:36.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:37.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:37.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:37.797 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:15:37.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:15:37.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:37.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:15:38.157 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/310034,v1:127.0.0.1:6827/310034] [v2:127.0.0.1:6828/310034,v1:127.0.0.1:6829/310034] exists,up a425b01d-eb5f-4429-85bd-8d8231ddc327 2021-07-26T07:15:38.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:38.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:38.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:38.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2021-07-26T07:15:38.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:15:38.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:38.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T07:15:38.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:38.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2021-07-26T07:15:38.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:15:38.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:38.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:38.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:38.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-07-26T07:15:38.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-07-26T07:15:38.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:38.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:38.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:38.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:15:38.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:38.164 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:38.165 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:38.165 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:38.165 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:38.165 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:15:38.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:38.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:38.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:38.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:38.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:38.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:15:38.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:15:38.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:38.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:38.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:15:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2021-07-26T07:15:38.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:38.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d8d6fb5-d072-4789-b53b-dc029dee5875 2021-07-26T07:15:38.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 1d8d6fb5-d072-4789-b53b-dc029dee5875' 2021-07-26T07:15:38.171 INFO:tasks.workunit.client.0.smithi094.stdout:add osd4 1d8d6fb5-d072-4789-b53b-dc029dee5875 2021-07-26T07:15:38.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:38.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaYf5gLUEWChAApNSIVZp77DfogkGyyApjUg== 2021-07-26T07:15:38.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaYf5gLUEWChAApNSIVZp77DfogkGyyApjUg=="}' 2021-07-26T07:15:38.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d8d6fb5-d072-4789-b53b-dc029dee5875 -i td/osd-recovery-stats/4/new.json 2021-07-26T07:15:38.710 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T07:15:38.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2021-07-26T07:15:38.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAaYf5gLUEWChAApNSIVZp77DfogkGyyApjUg== --osd-uuid 1d8d6fb5-d072-4789-b53b-dc029dee5875 2021-07-26T07:15:38.757 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:38.741+0000 7f3355efbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:38.760 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:38.745+0000 7f3355efbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:38.762 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:38.745+0000 7f3355efbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:38.763 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:38.749+0000 7f3355efbf40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:15:38.763 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:38.749+0000 7f3355efbf40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:15:38.764 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:38.749+0000 7f3355efbf40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:15:38.899 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:38.885+0000 7f3355efbf40 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-07-26T07:15:45.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-07-26T07:15:45.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:45.250 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd4 key to auth repository 2021-07-26T07:15:45.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T07:15:45.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:45.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T07:15:45.744 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.4 2021-07-26T07:15:45.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:45.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:45.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:45.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:45.769 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:45.753+0000 7f9d1bddff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:45.772 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:45.757+0000 7f9d1bddff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:45.775 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:45.757+0000 7f9d1bddff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:46.056 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:46.041+0000 7f9d1bddff40 -1 Falling back to public interface 2021-07-26T07:15:46.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T07:15:46.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:46.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T07:15:46.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:46.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:46.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:46.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:46.103 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:15:46.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:46.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:15:46.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:47.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:47.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:47.483 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:15:47.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:47.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:47.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:15:47.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:48.565 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:48.549+0000 7f9d1bddff40 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T07:15:48.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:48.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:48.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:48.861 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:15:48.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:48.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:15:49.399 INFO:tasks.workunit.client.0.smithi094.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/311440,v1:127.0.0.1:6835/311440] [v2:127.0.0.1:6836/311440,v1:127.0.0.1:6837/311440] exists,up 1d8d6fb5-d072-4789-b53b-dc029dee5875 2021-07-26T07:15:49.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:15:49.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:15:49.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:15:49.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2021-07-26T07:15:49.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:15:49.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:15:49.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T07:15:49.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:15:49.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2021-07-26T07:15:49.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:15:49.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:15:49.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:15:49.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:15:49.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-07-26T07:15:49.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-07-26T07:15:49.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:15:49.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:15:49.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:15:49.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:15:49.406 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:15:49.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:15:49.407 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:15:49.407 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:15:49.407 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:15:49.408 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:15:49.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:49.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:15:49.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:15:49.410 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:15:49.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:15:49.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:15:49.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:15:49.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:15:49.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:15:49.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:49.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:15:49.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:15:49.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2021-07-26T07:15:49.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:15:49.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e784bed4-0fe4-444c-8cc7-7bd471636000 2021-07-26T07:15:49.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 e784bed4-0fe4-444c-8cc7-7bd471636000' 2021-07-26T07:15:49.414 INFO:tasks.workunit.client.0.smithi094.stdout:add osd5 e784bed4-0fe4-444c-8cc7-7bd471636000 2021-07-26T07:15:49.414 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:15:49.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlYf5g4mjhGBAAIWtvrimtPqu9AuWZ8I4ZsA== 2021-07-26T07:15:49.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlYf5g4mjhGBAAIWtvrimtPqu9AuWZ8I4ZsA=="}' 2021-07-26T07:15:49.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e784bed4-0fe4-444c-8cc7-7bd471636000 -i td/osd-recovery-stats/5/new.json 2021-07-26T07:15:50.035 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T07:15:50.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2021-07-26T07:15:50.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAlYf5g4mjhGBAAIWtvrimtPqu9AuWZ8I4ZsA== --osd-uuid e784bed4-0fe4-444c-8cc7-7bd471636000 2021-07-26T07:15:50.083 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:50.069+0000 7f97b6217f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:50.090 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:50.073+0000 7f97b6217f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:50.092 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:50.077+0000 7f97b6217f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:50.093 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:50.077+0000 7f97b6217f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:15:50.094 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:50.077+0000 7f97b6217f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:15:50.094 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:50.077+0000 7f97b6217f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:15:50.190 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:50.172+0000 7f97b6217f40 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-07-26T07:15:56.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-07-26T07:15:56.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:15:56.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T07:15:56.335 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd5 key to auth repository 2021-07-26T07:15:56.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:15:56.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T07:15:56.818 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.5 2021-07-26T07:15:56.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:15:56.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:15:56.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:15:56.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:15:56.837 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:56.820+0000 7f21a6aa3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:56.839 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:56.824+0000 7f21a6aa3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:56.851 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:56.824+0000 7f21a6aa3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:15:57.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T07:15:57.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:15:57.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T07:15:57.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:15:57.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:15:57.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:57.171 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:15:57.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:15:57.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:57.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:15:57.380 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:57.364+0000 7f21a6aa3f40 -1 Falling back to public interface 2021-07-26T07:15:57.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:58.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:58.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:58.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:15:58.536 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:15:58.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:58.537 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:15:58.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:15:59.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:15:59.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:15:59.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:15:59.924 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:15:59.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:15:59.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:15:59.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:15:59.920+0000 7f21a6aa3f40 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T07:16:00.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:16:01.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:16:01.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:16:01.382 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:16:01.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:16:01.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:16:01.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:16:01.750 INFO:tasks.workunit.client.0.smithi094.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/312820,v1:127.0.0.1:6843/312820] [v2:127.0.0.1:6844/312820,v1:127.0.0.1:6845/312820] exists,up e784bed4-0fe4-444c-8cc7-7bd471636000 2021-07-26T07:16:01.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:16:01.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:16:01.751 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:16:01.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2021-07-26T07:16:01.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2021-07-26T07:16:01.752 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2021-07-26T07:16:02.401 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T07:16:02.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:16:03.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2021-07-26T07:16:03.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:16:03.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:16:03.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:16:03.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:16:03.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:16:03.422 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:16:03.422 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:16:03.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:16:03.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:16:03.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:16:03.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:16:03.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:16:03.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:16:03.635 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:16:03.635 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:16:04.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:16:04.017 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:16:04.017 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:16:04.017 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:16:04.017 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:16:04.017 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:16:04.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:16:04.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:04.018 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:16:04.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836495 2021-07-26T07:16:04.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836495 2021-07-26T07:16:04.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495' 2021-07-26T07:16:04.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:04.217 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:16:04.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672973 2021-07-26T07:16:04.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672973 2021-07-26T07:16:04.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672973' 2021-07-26T07:16:04.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:04.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:16:04.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509450 2021-07-26T07:16:04.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509450 2021-07-26T07:16:04.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450' 2021-07-26T07:16:04.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:04.505 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:16:04.643 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345928 2021-07-26T07:16:04.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345928 2021-07-26T07:16:04.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450 3-85899345928' 2021-07-26T07:16:04.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:04.645 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:16:04.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182405 2021-07-26T07:16:04.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182405 2021-07-26T07:16:04.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450 3-85899345928 4-107374182405' 2021-07-26T07:16:04.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:04.785 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:16:04.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018883 2021-07-26T07:16:04.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018883 2021-07-26T07:16:04.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509450 3-85899345928 4-107374182405 5-128849018883' 2021-07-26T07:16:04.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:04.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836495 2021-07-26T07:16:04.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:04.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:16:04.929 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836495 2021-07-26T07:16:04.929 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:04.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836495 2021-07-26T07:16:04.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-07-26T07:16:04.931 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836495 2021-07-26T07:16:04.932 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:16:05.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836493 -lt 21474836495 2021-07-26T07:16:05.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:16:06.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:16:06.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:16:06.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836495 2021-07-26T07:16:06.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:06.653 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:06.654 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672973 2021-07-26T07:16:06.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:16:06.656 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672973 2021-07-26T07:16:06.657 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:06.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672973 2021-07-26T07:16:06.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-07-26T07:16:06.659 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672973 2021-07-26T07:16:06.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:16:07.010 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672973 -lt 42949672973 2021-07-26T07:16:07.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:07.011 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509450 2021-07-26T07:16:07.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:07.013 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:16:07.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509450 2021-07-26T07:16:07.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:07.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509450 2021-07-26T07:16:07.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2021-07-26T07:16:07.017 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509450 2021-07-26T07:16:07.017 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:16:07.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509451 -lt 64424509450 2021-07-26T07:16:07.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:07.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345928 2021-07-26T07:16:07.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:07.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:16:07.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345928 2021-07-26T07:16:07.389 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:07.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345928 2021-07-26T07:16:07.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2021-07-26T07:16:07.391 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345928 2021-07-26T07:16:07.392 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:16:07.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345929 -lt 85899345928 2021-07-26T07:16:07.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:07.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182405 2021-07-26T07:16:07.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:07.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:16:07.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182405 2021-07-26T07:16:07.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:07.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182405 2021-07-26T07:16:07.827 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182405 2021-07-26T07:16:07.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-07-26T07:16:07.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:16:08.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182405 2021-07-26T07:16:08.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:08.186 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018883 2021-07-26T07:16:08.187 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:08.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:16:08.190 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018883 2021-07-26T07:16:08.190 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:08.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018883 2021-07-26T07:16:08.192 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-07-26T07:16:08.192 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018883 2021-07-26T07:16:08.193 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:16:08.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018884 -lt 128849018883 2021-07-26T07:16:08.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:16:08.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:08.548 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:09.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:16:09.060 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:09.060 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:09.060 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:09.060 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:09.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:09.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:09.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:09.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:16:09.426 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:09.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:09.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:09.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:16:09.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:16:09.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:16:09.906 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2021-07-26T07:16:09.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:09.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2021-07-26T07:16:10.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2021-07-26T07:16:10.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2021-07-26T07:16:10.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2021-07-26T07:16:10.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2021-07-26T07:16:10.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2021-07-26T07:16:10.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2021-07-26T07:16:10.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2021-07-26T07:16:10.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:10.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2021-07-26T07:16:11.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2021-07-26T07:16:11.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2021-07-26T07:16:11.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2021-07-26T07:16:11.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2021-07-26T07:16:11.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2021-07-26T07:16:11.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2021-07-26T07:16:11.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.796 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2021-07-26T07:16:11.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2021-07-26T07:16:11.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:11.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2021-07-26T07:16:12.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:12.264 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2021-07-26T07:16:12.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:12.390 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2021-07-26T07:16:12.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:12.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2021-07-26T07:16:12.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:12.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2021-07-26T07:16:12.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:12.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2021-07-26T07:16:12.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:12.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2021-07-26T07:16:13.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2021-07-26T07:16:13.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2021-07-26T07:16:13.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2021-07-26T07:16:13.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2021-07-26T07:16:13.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2021-07-26T07:16:13.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2021-07-26T07:16:13.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2021-07-26T07:16:13.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:13.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2021-07-26T07:16:14.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2021-07-26T07:16:14.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2021-07-26T07:16:14.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2021-07-26T07:16:14.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2021-07-26T07:16:14.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2021-07-26T07:16:14.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2021-07-26T07:16:14.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2021-07-26T07:16:14.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2021-07-26T07:16:14.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:14.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2021-07-26T07:16:15.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2021-07-26T07:16:15.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2021-07-26T07:16:15.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2021-07-26T07:16:15.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2021-07-26T07:16:15.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2021-07-26T07:16:15.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.596 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2021-07-26T07:16:15.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2021-07-26T07:16:15.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2021-07-26T07:16:15.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:15.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2021-07-26T07:16:16.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2021-07-26T07:16:16.164 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2021-07-26T07:16:16.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2021-07-26T07:16:16.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.433 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2021-07-26T07:16:16.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2021-07-26T07:16:16.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2021-07-26T07:16:16.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2021-07-26T07:16:16.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:16.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2021-07-26T07:16:17.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2021-07-26T07:16:17.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.194 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2021-07-26T07:16:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2021-07-26T07:16:17.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2021-07-26T07:16:17.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.554 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2021-07-26T07:16:17.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2021-07-26T07:16:17.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2021-07-26T07:16:17.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2021-07-26T07:16:17.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:17.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2021-07-26T07:16:18.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2021-07-26T07:16:18.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2021-07-26T07:16:18.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2021-07-26T07:16:18.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2021-07-26T07:16:18.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2021-07-26T07:16:18.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2021-07-26T07:16:18.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2021-07-26T07:16:18.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:18.960 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2021-07-26T07:16:19.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2021-07-26T07:16:19.278 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2021-07-26T07:16:19.403 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2021-07-26T07:16:19.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2021-07-26T07:16:19.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2021-07-26T07:16:19.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2021-07-26T07:16:19.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2021-07-26T07:16:19.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:19.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2021-07-26T07:16:20.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2021-07-26T07:16:20.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2021-07-26T07:16:20.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2021-07-26T07:16:20.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2021-07-26T07:16:20.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2021-07-26T07:16:20.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2021-07-26T07:16:20.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2021-07-26T07:16:20.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:20.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2021-07-26T07:16:21.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2021-07-26T07:16:21.222 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2021-07-26T07:16:21.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2021-07-26T07:16:21.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2021-07-26T07:16:21.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2021-07-26T07:16:21.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2021-07-26T07:16:21.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2021-07-26T07:16:21.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:21.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2021-07-26T07:16:22.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2021-07-26T07:16:22.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2021-07-26T07:16:22.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2021-07-26T07:16:22.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2021-07-26T07:16:22.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2021-07-26T07:16:22.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2021-07-26T07:16:22.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2021-07-26T07:16:22.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:22.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2021-07-26T07:16:23.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2021-07-26T07:16:23.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.107 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2021-07-26T07:16:23.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2021-07-26T07:16:23.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2021-07-26T07:16:23.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2021-07-26T07:16:23.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2021-07-26T07:16:23.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.651 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2021-07-26T07:16:23.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2021-07-26T07:16:23.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2021-07-26T07:16:23.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:23.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2021-07-26T07:16:24.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:24.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2021-07-26T07:16:24.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:24.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2021-07-26T07:16:24.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:24.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2021-07-26T07:16:24.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:24.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2021-07-26T07:16:24.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:24.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2021-07-26T07:16:24.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:24.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2021-07-26T07:16:24.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:24.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2021-07-26T07:16:25.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2021-07-26T07:16:25.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2021-07-26T07:16:25.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2021-07-26T07:16:25.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2021-07-26T07:16:25.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2021-07-26T07:16:25.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2021-07-26T07:16:25.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2021-07-26T07:16:25.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:25.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2021-07-26T07:16:26.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2021-07-26T07:16:26.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2021-07-26T07:16:26.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2021-07-26T07:16:26.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2021-07-26T07:16:26.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2021-07-26T07:16:26.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2021-07-26T07:16:26.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2021-07-26T07:16:26.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:26.908 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2021-07-26T07:16:27.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2021-07-26T07:16:27.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2021-07-26T07:16:27.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2021-07-26T07:16:27.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2021-07-26T07:16:27.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2021-07-26T07:16:27.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2021-07-26T07:16:27.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2021-07-26T07:16:27.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2021-07-26T07:16:27.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:27.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2021-07-26T07:16:28.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2021-07-26T07:16:28.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2021-07-26T07:16:28.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2021-07-26T07:16:28.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2021-07-26T07:16:28.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2021-07-26T07:16:28.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2021-07-26T07:16:28.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2021-07-26T07:16:28.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:28.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2021-07-26T07:16:29.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2021-07-26T07:16:29.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2021-07-26T07:16:29.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2021-07-26T07:16:29.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2021-07-26T07:16:29.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.530 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2021-07-26T07:16:29.638 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2021-07-26T07:16:29.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2021-07-26T07:16:29.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2021-07-26T07:16:29.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:29.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2021-07-26T07:16:30.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2021-07-26T07:16:30.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2021-07-26T07:16:30.333 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2021-07-26T07:16:30.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2021-07-26T07:16:30.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2021-07-26T07:16:30.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2021-07-26T07:16:30.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2021-07-26T07:16:30.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:30.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2021-07-26T07:16:31.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2021-07-26T07:16:31.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.179 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2021-07-26T07:16:31.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2021-07-26T07:16:31.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2021-07-26T07:16:31.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2021-07-26T07:16:31.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2021-07-26T07:16:31.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2021-07-26T07:16:31.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2021-07-26T07:16:31.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:31.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2021-07-26T07:16:32.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2021-07-26T07:16:32.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2021-07-26T07:16:32.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2021-07-26T07:16:32.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2021-07-26T07:16:32.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2021-07-26T07:16:32.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.678 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2021-07-26T07:16:32.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2021-07-26T07:16:32.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:32.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2021-07-26T07:16:33.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2021-07-26T07:16:33.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2021-07-26T07:16:33.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2021-07-26T07:16:33.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.415 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2021-07-26T07:16:33.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2021-07-26T07:16:33.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2021-07-26T07:16:33.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.733 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2021-07-26T07:16:33.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:33.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2021-07-26T07:16:34.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-07-26T07:16:34.001 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2021-07-26T07:16:34.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2021-07-26T07:16:34.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:16:34.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:16:34.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:16:34.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:16:34.484 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2021-07-26T07:16:34.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2021-07-26T07:16:34.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T07:16:34.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:16:34.486 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test obj1 2021-07-26T07:16:34.487 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:16:34.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2021-07-26T07:16:34.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2021-07-26T07:16:34.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test 2021-07-26T07:16:34.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T07:16:34.861 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test obj1 2021-07-26T07:16:34.861 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:16:34.862 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:16:34.862 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:16:34.862 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:16:35.262 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:16:35.263 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:16:35.263 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test obj1 2021-07-26T07:16:35.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2021-07-26T07:16:35.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2021-07-26T07:16:36.398 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is set 2021-07-26T07:16:36.411 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2021-07-26T07:16:36.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 305813 2021-07-26T07:16:36.413 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2021-07-26T07:16:36.943 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already down. 2021-07-26T07:16:36.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2021-07-26T07:16:37.714 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T07:16:37.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2021-07-26T07:16:38.443 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is unset 2021-07-26T07:16:38.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2021-07-26T07:16:38.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:16:38.456 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:16:38.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:16:38.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:16:39.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2021-07-26T07:16:39.237 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T07:16:39.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2021-07-26T07:16:41.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2021-07-26T07:16:41.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:16:41.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:16:41.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:16:41.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:16:41.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:16:41.250 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:16:41.251 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:16:41.251 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:16:41.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:16:41.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:16:41.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:16:41.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:16:41.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:16:41.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:16:41.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:16:41.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:16:41.830 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:16:41.831 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:16:41.831 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:16:41.831 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:16:41.831 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:16:41.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:16:41.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:41.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:16:41.931 INFO:tasks.workunit.client.0.smithi094.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-07-26T07:16:41.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq= 2021-07-26T07:16:41.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z '' 2021-07-26T07:16:41.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: continue 2021-07-26T07:16:41.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:41.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:16:42.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672982 2021-07-26T07:16:42.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672982 2021-07-26T07:16:42.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672982' 2021-07-26T07:16:42.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:42.170 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:16:42.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509460 2021-07-26T07:16:42.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509460 2021-07-26T07:16:42.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672982 2-64424509460' 2021-07-26T07:16:42.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:42.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:16:42.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345937 2021-07-26T07:16:42.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345937 2021-07-26T07:16:42.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672982 2-64424509460 3-85899345937' 2021-07-26T07:16:42.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:42.469 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:16:42.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182415 2021-07-26T07:16:42.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182415 2021-07-26T07:16:42.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672982 2-64424509460 3-85899345937 4-107374182415' 2021-07-26T07:16:42.615 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:16:42.616 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:16:42.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018893 2021-07-26T07:16:42.763 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018893 2021-07-26T07:16:42.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 1-42949672982 2-64424509460 3-85899345937 4-107374182415 5-128849018893' 2021-07-26T07:16:42.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:42.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672982 2021-07-26T07:16:42.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:42.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:16:42.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672982 2021-07-26T07:16:42.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:42.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672982 2021-07-26T07:16:42.769 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672982 2021-07-26T07:16:42.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2021-07-26T07:16:42.770 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:16:43.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672980 -lt 42949672982 2021-07-26T07:16:43.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:16:44.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:16:44.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:16:44.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672983 -lt 42949672982 2021-07-26T07:16:44.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:44.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509460 2021-07-26T07:16:44.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:44.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:16:44.613 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509460 2021-07-26T07:16:44.613 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:44.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509460 2021-07-26T07:16:44.617 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509460 2021-07-26T07:16:44.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2021-07-26T07:16:44.618 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:16:45.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509460 -lt 64424509460 2021-07-26T07:16:45.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:45.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345937 2021-07-26T07:16:45.005 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:45.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:16:45.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345937 2021-07-26T07:16:45.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:45.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345937 2021-07-26T07:16:45.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2021-07-26T07:16:45.011 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345937 2021-07-26T07:16:45.012 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:16:45.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345938 -lt 85899345937 2021-07-26T07:16:45.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:45.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182415 2021-07-26T07:16:45.417 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:45.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:16:45.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182415 2021-07-26T07:16:45.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:45.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182415 2021-07-26T07:16:45.423 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182415 2021-07-26T07:16:45.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2021-07-26T07:16:45.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:16:45.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182416 -lt 107374182415 2021-07-26T07:16:45.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:16:45.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:16:45.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018893 2021-07-26T07:16:45.816 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:16:45.817 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018893 2021-07-26T07:16:45.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:16:45.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018893 2021-07-26T07:16:45.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2021-07-26T07:16:45.820 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018893 2021-07-26T07:16:45.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:16:46.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018893 -lt 128849018893 2021-07-26T07:16:46.228 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:16:46.229 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:46.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:46.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:16:46.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:46.771 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:46.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:46.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:46.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:46.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:46.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:47.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:47.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:47.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:47.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:47.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:47.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:16:47.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:16:47.649 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:16:47.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:47.650 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:47.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:47.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:47.751 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:47.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:47.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:47.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:47.752 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:47.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:48.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:48.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:48.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:48.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:48.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:48.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:48.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:48.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:48.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:48.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:48.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:48.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:48.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:49.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:16:49.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:16:49.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:49.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:49.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:49.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:49.230 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:49.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:49.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:49.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:49.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:49.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:49.232 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:49.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:49.563 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:49.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:49.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:50.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:50.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:50.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:50.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:50.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:50.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:50.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:50.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:50.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:50.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:16:50.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:16:50.603 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:50.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:50.604 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:50.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:50.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:50.705 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:50.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:50.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:50.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:50.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:50.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:51.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:51.057 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:51.058 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:51.058 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:51.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:51.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:51.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:51.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:51.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:51.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:51.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:51.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:51.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:51.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:16:51.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:16:51.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:51.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:51.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:52.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:52.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:52.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:52.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:52.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:52.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:52.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:52.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:52.514 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:52.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:52.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:52.515 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:52.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:52.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:52.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:52.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:52.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:52.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:52.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:52.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:52.981 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:53.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:16:53.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:16:53.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:53.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:53.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:53.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:53.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:53.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:53.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:53.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:53.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:53.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:53.626 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:53.962 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:53.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:53.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:53.963 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:54.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:54.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:54.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:54.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:54.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:54.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:54.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:54.408 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:54.408 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:54.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=6 2021-07-26T07:16:54.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 6 '!=' null 2021-07-26T07:16:54.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:54.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:54.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:55.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:55.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:55.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:55.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:55.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:55.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:55.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:55.037 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:55.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:55.458 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:55.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:55.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:55.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:55.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:55.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:55.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:55.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:55.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:55.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:55.923 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:55.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:56.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:16:56.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:16:56.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:56.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:56.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:56.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:56.461 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:56.462 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:56.462 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:56.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:56.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:56.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:56.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:56.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:56.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:56.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:56.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:57.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:57.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:57.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:57.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:57.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:57.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:57.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:57.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:57.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:57.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:16:57.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:16:57.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:57.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:57.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:57.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:57.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:57.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:57.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:57.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:57.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:57.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:57.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:58.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:58.257 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:58.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:58.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:16:58.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:16:58.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:16:58.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:16:58.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:16:58.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:16:58.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:16:58.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:16:58.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:16:58.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:16:59.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:16:59.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:16:59.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:16:59.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:16:59.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:16:59.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:16:59.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:16:59.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:16:59.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:16:59.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:16:59.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:16:59.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:16:59.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:16:59.637 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:16:59.637 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:16:59.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:16:59.638 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:00.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:00.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:00.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:00.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:00.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:00.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:00.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:00.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:00.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:00.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:17:00.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:17:00.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:00.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:00.602 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:00.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:00.702 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:00.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:00.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:00.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:00.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:00.704 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:00.705 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:01.042 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:01.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:01.043 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:01.044 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:01.531 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:01.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:01.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:01.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:01.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:01.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:01.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:01.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:01.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:02.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:17:02.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:17:02.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:02.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:02.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:02.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:02.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:02.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:02.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:02.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:02.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:02.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:02.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:02.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:02.509 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:02.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:02.510 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:03.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:03.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:03.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:03.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:03.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:03.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:03.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:03.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:03.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:03.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:17:03.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:17:03.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:03.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:03.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:03.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:03.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:03.723 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:03.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:03.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:03.724 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:03.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:03.725 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:04.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:04.076 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:04.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:04.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:04.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:04.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:04.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:04.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:04.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:04.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:04.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:04.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:04.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:05.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:17:05.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:17:05.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:05.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:05.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:05.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:05.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:05.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:05.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:05.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:05.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:05.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:05.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:05.489 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:05.490 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:05.491 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:05.491 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:06.094 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:06.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:06.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:06.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:06.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:06.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:06.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:06.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:06.097 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:06.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:17:06.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:17:06.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:06.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:06.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:06.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:06.672 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:06.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:06.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:06.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:06.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:06.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:06.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:07.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:07.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:07.038 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:07.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:07.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:07.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:07.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:07.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:07.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:07.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:07.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:07.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:07.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:08.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:17:08.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:17:08.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:08.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:08.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:08.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:08.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:08.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:08.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:08.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:08.135 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:08.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:08.136 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:08.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:08.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:08.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:08.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:09.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:09.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:09.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:09.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:09.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:09.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:09.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:09.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:09.035 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:09.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:17:09.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:17:09.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:09.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:09.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:09.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:09.679 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:09.680 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:09.680 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:09.680 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:09.681 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:09.681 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:09.681 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:10.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:10.015 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:10.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:10.016 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:10.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:10.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:10.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:10.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:10.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:10.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:10.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:10.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:10.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:11.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=7 2021-07-26T07:17:11.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 7 '!=' null 2021-07-26T07:17:11.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:11.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:11.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:11.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:11.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:11.141 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:11.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:11.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:11.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:11.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:11.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:11.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:11.495 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:11.495 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:11.496 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:11.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:11.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:11.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:11.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:11.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:11.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:11.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:11.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:11.975 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:12.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:17:12.434 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:17:12.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:12.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:12.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:12.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:12.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:12.536 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:12.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:12.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:12.537 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:12.538 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:12.538 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:12.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:17:12.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:12.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:12.941 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:13.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:17:13.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:17:13.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:17:13.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:17:13.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:17:13.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:17:13.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:17:13.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:17:13.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:17:13.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:17:13.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:17:13.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:17:13.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:17:13.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:17:14.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:17:14.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:17:14.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:17:14.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:17:14.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:17:14.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:17:14.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:17:14.051 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:17:14.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:17:14.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:17:14.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:17:14.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:17:14.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:17:14.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:17:14.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:17:14.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2021-07-26T07:17:14.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-07-26T07:17:14.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-07-26T07:17:14.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2021-07-26T07:17:14.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-07-26T07:17:14.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2021-07-26T07:17:14.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-07-26T07:17:14.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2021-07-26T07:17:14.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-07-26T07:17:14.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-07-26T07:17:14.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-07-26T07:17:14.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2021-07-26T07:17:14.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-07-26T07:17:14.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-07-26T07:17:14.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-07-26T07:17:14.828 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:17:14.828 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-07-26T07:17:14.828 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-07-26T07:17:14.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2021-07-26T07:17:14.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-07-26T07:17:14.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:17:14.840 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:17:14.841 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2021-07-26T07:17:14.841 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-07-26T07:17:14.841 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-07-26T07:17:14.841 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:17:14.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2021-07-26T07:17:14.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2021-07-26T07:17:14.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-07-26T07:17:14.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:17:14.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-07-26T07:17:14.852 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:17:14.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:17:14.853 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-07-26T07:17:14.853 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:17:14.853 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:17:14.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-07-26T07:17:14.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-07-26T07:17:14.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:17:14.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:17:14.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:17:14.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:17:14.868 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-07-26T07:17:14.869 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:17:14.870 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:17:14.870 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2021-07-26T07:17:14.870 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-07-26T07:17:14.870 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-07-26T07:17:14.871 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:17:14.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2021-07-26T07:17:14.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2021-07-26T07:17:14.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2021-07-26T07:17:14.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:17:14.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2021-07-26T07:17:14.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:17:14.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:17:14.876 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-07-26T07:17:14.876 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-07-26T07:17:14.877 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:17:14.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-07-26T07:17:14.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-07-26T07:17:14.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:17:14.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:17:14.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:17:14.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:17:14.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-07-26T07:17:14.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2021-07-26T07:17:14.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T07:17:14.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T07:17:15.368 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T07:17:15.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2021-07-26T07:17:15.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:17:15.383 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:17:15.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:17:15.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:17:15.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:17:15.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:17:15.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-07-26T07:17:15.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:17:15.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:17:15.503 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:17:15.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:17:15.504 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:17:15.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:17:15.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:17:15.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:17:15.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:17:15.518 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:17:15.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:17:15.520 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:17:15.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:17:15.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:17:15.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:17:15.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:15.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:17:15.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:17:15.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:15.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:17:15.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:17:15.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:17:15.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:17:15.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:15.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:15.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:17:15.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:17:15.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:17:15.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-07-26T07:17:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-07-26T07:17:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-07-26T07:17:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-07-26T07:17:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:17:15.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:17:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:17:15.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:17:15.550 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:17:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:17:15.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:17:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:17:15.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:17:15.552 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:17:15.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:17:15.553 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:17:15.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:17:15.555 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:17:15.556 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:17:15.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:15.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:17:15.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:17:15.558 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:17:15.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:17:15.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:17:15.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:17:15.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:17:15.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:15.564 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:15.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:17:15.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:17:15.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:17:15.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-07-26T07:17:15.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:17:15.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:15.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:15.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257957 2021-07-26T07:17:15.571 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:17:15.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:17:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:17:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-07-26T07:17:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2021-07-26T07:17:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2021-07-26T07:17:15.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2021-07-26T07:17:15.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-07-26T07:17:15.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:17:15.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:17:15.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:17:15.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-07-26T07:17:15.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-07-26T07:17:15.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:17:15.953 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:15.954 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:15.955 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:15.955 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:15.955 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:15.956 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:15.956 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:17:16.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:17:16.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:17:16.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:17:16.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:17:16.246 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:17:16.247 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:17:16.247 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:17:16.247 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:17:16.247 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:17:16.248 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:17:16.253 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:16.253 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:16.253 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:17:16.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:17:16.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get fsid 2021-07-26T07:17:16.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:17:16.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:17:16.350 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:17:16.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:17:16.351 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:17:16.352 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:17:16.352 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:17:16.352 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:17:16.353 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:17:16.353 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:16.353 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:16.353 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:17:16.354 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:17:16.354 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get mon_host 2021-07-26T07:17:16.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2021-07-26T07:17:16.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2021-07-26T07:17:16.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:17:16.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:17:16.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:17:16.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2021-07-26T07:17:16.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:17:17.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:17:17.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:17.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:17.332 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:17.332 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:17.333 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:17.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:17.333 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:17:17.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:17:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2021-07-26T07:17:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:17:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:17:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:17:17.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:17:17.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:17:17.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:17:17.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:17:17.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:17:17.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:17:17.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:17:17.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:17:17.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:17:17.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:17:17.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:17:17.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:17:17.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:17:17.372 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:17.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:17.374 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:17.375 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:17.375 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:17.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:17.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:17:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:17:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:17:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:17:17.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:17:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:17:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:17:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:17:17.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:17:17.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:17:17.378 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:17:17.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f6a9b42a-63ab-40f3-b4e6-56f572eeab99 2021-07-26T07:17:17.379 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 f6a9b42a-63ab-40f3-b4e6-56f572eeab99 2021-07-26T07:17:17.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f6a9b42a-63ab-40f3-b4e6-56f572eeab99' 2021-07-26T07:17:17.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:17:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9Yf5gvMW3FhAA7hWFFL80e7Zrt5/wEyrfhw== 2021-07-26T07:17:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9Yf5gvMW3FhAA7hWFFL80e7Zrt5/wEyrfhw=="}' 2021-07-26T07:17:17.393 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f6a9b42a-63ab-40f3-b4e6-56f572eeab99 -i td/osd-recovery-stats/0/new.json 2021-07-26T07:17:17.743 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:17:17.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2021-07-26T07:17:17.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB9Yf5gvMW3FhAA7hWFFL80e7Zrt5/wEyrfhw== --osd-uuid f6a9b42a-63ab-40f3-b4e6-56f572eeab99 2021-07-26T07:17:17.789 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:17.775+0000 7f5cdca26f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:17.792 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:17.779+0000 7f5cdca26f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:17.794 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:17.779+0000 7f5cdca26f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:17.795 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:17.779+0000 7f5cdca26f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:17:17.795 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:17.779+0000 7f5cdca26f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:17:17.795 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:17.779+0000 7f5cdca26f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:17:17.872 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:17.859+0000 7f5cdca26f40 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-07-26T07:17:23.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-07-26T07:17:23.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:17:23.470 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:17:23.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:17:23.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:17:23.940 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:17:23.941 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:17:23.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:23.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:17:23.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:17:23.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:17:23.970 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:23.955+0000 7f387267af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:23.972 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:23.959+0000 7f387267af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:23.973 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:23.959+0000 7f387267af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:24.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:17:24.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:17:24.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:17:24.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:17:24.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:17:24.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:24.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:17:24.286 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:17:24.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:24.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:24.517 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:24.503+0000 7f387267af40 -1 Falling back to public interface 2021-07-26T07:17:24.661 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:25.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:25.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:25.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:17:25.664 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:17:25.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:25.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:26.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:26.884 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:26.871+0000 7f387267af40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:17:27.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:27.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:27.015 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:17:27.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:17:27.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:27.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:27.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:28.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:28.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:28.471 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:17:28.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:17:28.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:28.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:17:28.836 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/321444,v1:127.0.0.1:6803/321444] [v2:127.0.0.1:6804/321444,v1:127.0.0.1:6805/321444] exists,up f6a9b42a-63ab-40f3-b4e6-56f572eeab99 2021-07-26T07:17:28.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:17:28.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:17:28.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:17:28.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2021-07-26T07:17:28.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:17:28.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:17:28.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:17:28.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:17:28.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:17:28.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:17:28.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:17:28.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:17:28.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:17:28.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:17:28.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:17:28.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:17:28.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:17:28.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:17:28.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:17:28.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:17:28.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:28.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:28.845 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:28.846 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:28.846 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:28.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:28.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:28.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:17:28.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:17:28.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:17:28.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:17:28.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:17:28.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:17:28.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:17:28.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:28.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:17:28.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:17:28.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:17:28.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:17:28.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a486a16f-c583-4b4b-9992-738e2a10ae60 2021-07-26T07:17:28.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a486a16f-c583-4b4b-9992-738e2a10ae60' 2021-07-26T07:17:28.851 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 a486a16f-c583-4b4b-9992-738e2a10ae60 2021-07-26T07:17:28.851 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:17:28.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIYf5gRRqfMhAACT1BjVn9g+4Pf3NU9AhKpw== 2021-07-26T07:17:28.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIYf5gRRqfMhAACT1BjVn9g+4Pf3NU9AhKpw=="}' 2021-07-26T07:17:28.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a486a16f-c583-4b4b-9992-738e2a10ae60 -i td/osd-recovery-stats/1/new.json 2021-07-26T07:17:29.359 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:17:29.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2021-07-26T07:17:29.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCIYf5gRRqfMhAACT1BjVn9g+4Pf3NU9AhKpw== --osd-uuid a486a16f-c583-4b4b-9992-738e2a10ae60 2021-07-26T07:17:29.406 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:29.391+0000 7f01d75faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:29.409 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:29.395+0000 7f01d75faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:29.411 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:29.399+0000 7f01d75faf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:29.411 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:29.399+0000 7f01d75faf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:17:29.412 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:29.399+0000 7f01d75faf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:17:29.412 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:29.399+0000 7f01d75faf40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:17:29.502 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:29.487+0000 7f01d75faf40 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-07-26T07:17:35.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-07-26T07:17:35.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:17:35.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:17:35.670 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:17:35.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:17:36.173 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:17:36.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:17:36.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:36.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:17:36.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:17:36.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:17:36.202 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:36.187+0000 7f950f46cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:36.204 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:36.191+0000 7f950f46cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:36.205 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:36.191+0000 7f950f46cf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:36.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:17:36.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:17:36.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:17:36.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:17:36.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:17:36.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:36.562 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:17:36.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:17:36.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:36.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:17:36.741 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:36.727+0000 7f950f46cf40 -1 Falling back to public interface 2021-07-26T07:17:36.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:37.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:37.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:37.943 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:17:37.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:17:37.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:37.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:17:38.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:39.228 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:39.215+0000 7f950f46cf40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:17:39.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:39.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:39.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:17:39.305 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:17:39.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:39.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:17:39.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:40.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:17:40.854 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:17:40.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:40.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:17:41.212 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/322850,v1:127.0.0.1:6811/322850] [v2:127.0.0.1:6812/322850,v1:127.0.0.1:6813/322850] exists,up a486a16f-c583-4b4b-9992-738e2a10ae60 2021-07-26T07:17:41.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:17:41.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:17:41.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:17:41.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2021-07-26T07:17:41.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:17:41.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:17:41.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:17:41.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:17:41.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2021-07-26T07:17:41.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:17:41.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:17:41.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:17:41.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:17:41.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-07-26T07:17:41.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-07-26T07:17:41.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:17:41.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:17:41.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:17:41.221 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:17:41.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:17:41.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:41.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:41.222 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:41.222 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:41.222 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:41.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:41.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:41.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:17:41.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:17:41.223 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:17:41.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:17:41.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:17:41.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:17:41.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:17:41.224 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:41.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:17:41.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:17:41.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2021-07-26T07:17:41.226 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:17:41.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=714a668f-a157-4753-8b4e-e635c0e2712a 2021-07-26T07:17:41.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 714a668f-a157-4753-8b4e-e635c0e2712a' 2021-07-26T07:17:41.227 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 714a668f-a157-4753-8b4e-e635c0e2712a 2021-07-26T07:17:41.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:17:41.237 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVYf5gXRB1DRAAsHzEcB8DlUVWihPb21UUDQ== 2021-07-26T07:17:41.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVYf5gXRB1DRAAsHzEcB8DlUVWihPb21UUDQ=="}' 2021-07-26T07:17:41.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 714a668f-a157-4753-8b4e-e635c0e2712a -i td/osd-recovery-stats/2/new.json 2021-07-26T07:17:41.836 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:17:41.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2021-07-26T07:17:41.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCVYf5gXRB1DRAAsHzEcB8DlUVWihPb21UUDQ== --osd-uuid 714a668f-a157-4753-8b4e-e635c0e2712a 2021-07-26T07:17:41.877 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:41.863+0000 7f1e03e9ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:41.892 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:41.879+0000 7f1e03e9ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:41.894 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:41.883+0000 7f1e03e9ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:41.895 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:41.883+0000 7f1e03e9ff40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:17:41.895 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:41.883+0000 7f1e03e9ff40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:17:41.896 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:41.883+0000 7f1e03e9ff40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:17:41.964 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:41.951+0000 7f1e03e9ff40 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-07-26T07:17:48.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-07-26T07:17:48.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:17:48.450 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T07:17:48.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:17:48.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:17:48.947 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T07:17:48.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:17:48.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:17:48.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:48.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:17:48.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:17:48.975 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:48.963+0000 7fde670a6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:48.978 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:48.967+0000 7fde670a6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:48.980 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:48.967+0000 7fde670a6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:49.269 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:49.255+0000 7fde670a6f40 -1 Falling back to public interface 2021-07-26T07:17:49.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:17:49.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:17:49.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:17:49.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:17:49.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:17:49.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:49.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:17:49.356 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:17:49.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:49.357 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:17:49.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:50.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:50.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:50.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:17:50.709 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:17:50.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:50.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:17:51.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:51.877 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:51.863+0000 7fde670a6f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:17:52.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:52.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:52.068 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:17:52.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:17:52.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:52.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:17:52.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:17:53.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:17:53.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:17:53.574 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:17:53.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:17:53.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:17:53.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:17:53.918 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/324256,v1:127.0.0.1:6819/324256] [v2:127.0.0.1:6820/324256,v1:127.0.0.1:6821/324256] exists,up 714a668f-a157-4753-8b4e-e635c0e2712a 2021-07-26T07:17:53.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:17:53.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:17:53.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:17:53.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2021-07-26T07:17:53.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:17:53.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:17:53.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T07:17:53.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:17:53.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2021-07-26T07:17:53.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:17:53.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:17:53.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:17:53.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:17:53.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-07-26T07:17:53.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-07-26T07:17:53.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:17:53.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:17:53.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:17:53.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:17:53.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:17:53.928 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:17:53.928 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:17:53.928 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:17:53.928 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:17:53.928 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:17:53.929 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:53.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:17:53.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:17:53.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:17:53.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:17:53.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:17:53.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:17:53.930 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:17:53.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:17:53.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:17:53.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:17:53.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:17:53.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2021-07-26T07:17:53.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:17:53.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cffb08f1-eb50-40b5-ab86-3b793cc100fe 2021-07-26T07:17:53.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 cffb08f1-eb50-40b5-ab86-3b793cc100fe' 2021-07-26T07:17:53.934 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 cffb08f1-eb50-40b5-ab86-3b793cc100fe 2021-07-26T07:17:53.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:17:53.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChYf5giza+NxAAqjOvIyUDugOtF1wXNVVrpg== 2021-07-26T07:17:53.946 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChYf5giza+NxAAqjOvIyUDugOtF1wXNVVrpg=="}' 2021-07-26T07:17:53.947 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cffb08f1-eb50-40b5-ab86-3b793cc100fe -i td/osd-recovery-stats/3/new.json 2021-07-26T07:17:54.554 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:17:54.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2021-07-26T07:17:54.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQChYf5giza+NxAAqjOvIyUDugOtF1wXNVVrpg== --osd-uuid cffb08f1-eb50-40b5-ab86-3b793cc100fe 2021-07-26T07:17:54.605 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:54.591+0000 7fb29abf3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:54.660 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:54.647+0000 7fb29abf3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:54.662 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:54.651+0000 7fb29abf3f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:17:54.664 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:54.651+0000 7fb29abf3f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:17:54.664 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:54.651+0000 7fb29abf3f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:17:54.664 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:54.651+0000 7fb29abf3f40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:17:54.759 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:17:54.747+0000 7fb29abf3f40 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-07-26T07:18:00.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-07-26T07:18:00.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:18:00.970 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T07:18:00.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T07:18:00.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:18:01.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T07:18:01.456 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T07:18:01.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:18:01.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:18:01.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:18:01.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:18:01.477 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:01.463+0000 7f2ec66aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:01.479 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:01.467+0000 7f2ec66aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:01.482 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:01.471+0000 7f2ec66aaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:01.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T07:18:01.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:18:01.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:18:01.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:18:01.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:18:01.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:01.889 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:18:01.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:18:01.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:01.890 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:18:02.040 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:02.027+0000 7f2ec66aaf40 -1 Falling back to public interface 2021-07-26T07:18:02.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:03.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:03.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:03.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:18:03.252 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:18:03.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:03.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:18:03.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:04.554 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:04.539+0000 7f2ec66aaf40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:18:04.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:04.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:04.607 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:18:04.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:18:04.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:04.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:18:05.185 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:06.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:06.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:06.188 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:18:06.189 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:18:06.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:06.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:18:06.557 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/325662,v1:127.0.0.1:6827/325662] [v2:127.0.0.1:6828/325662,v1:127.0.0.1:6829/325662] exists,up cffb08f1-eb50-40b5-ab86-3b793cc100fe 2021-07-26T07:18:06.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:18:06.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:18:06.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:18:06.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2021-07-26T07:18:06.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:18:06.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:18:06.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T07:18:06.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:18:06.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2021-07-26T07:18:06.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:18:06.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:18:06.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:18:06.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:18:06.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-07-26T07:18:06.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-07-26T07:18:06.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:18:06.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:18:06.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:18:06.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:18:06.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:18:06.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:18:06.566 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:18:06.566 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:18:06.567 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:06.567 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:18:06.567 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:18:06.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:18:06.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:18:06.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:18:06.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:18:06.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:18:06.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:18:06.568 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:18:06.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:18:06.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:18:06.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:18:06.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:18:06.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2021-07-26T07:18:06.570 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:18:06.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d431205-93fd-4781-be34-fa43af3a6a46 2021-07-26T07:18:06.571 INFO:tasks.workunit.client.0.smithi094.stdout:add osd4 2d431205-93fd-4781-be34-fa43af3a6a46 2021-07-26T07:18:06.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 2d431205-93fd-4781-be34-fa43af3a6a46' 2021-07-26T07:18:06.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:18:06.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuYf5g4kIfIhAAJWKCQDrKPbcULzlvDVyqxw== 2021-07-26T07:18:06.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuYf5g4kIfIhAAJWKCQDrKPbcULzlvDVyqxw=="}' 2021-07-26T07:18:06.584 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d431205-93fd-4781-be34-fa43af3a6a46 -i td/osd-recovery-stats/4/new.json 2021-07-26T07:18:07.085 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T07:18:07.097 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2021-07-26T07:18:07.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCuYf5g4kIfIhAAJWKCQDrKPbcULzlvDVyqxw== --osd-uuid 2d431205-93fd-4781-be34-fa43af3a6a46 2021-07-26T07:18:07.126 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:07.115+0000 7f3102277f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:07.137 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:07.123+0000 7f3102277f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:07.139 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:07.127+0000 7f3102277f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:07.140 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:07.127+0000 7f3102277f40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:18:07.140 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:07.127+0000 7f3102277f40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:18:07.140 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:07.127+0000 7f3102277f40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:18:07.183 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:07.171+0000 7f3102277f40 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-07-26T07:18:13.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-07-26T07:18:13.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:18:13.289 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd4 key to auth repository 2021-07-26T07:18:13.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T07:18:13.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:18:13.785 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.4 2021-07-26T07:18:13.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T07:18:13.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:18:13.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:18:13.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:18:13.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:18:13.814 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:13.798+0000 7fad4e0e6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:13.816 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:13.802+0000 7fad4e0e6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:13.818 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:13.802+0000 7fad4e0e6f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:14.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T07:18:14.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:18:14.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T07:18:14.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:18:14.134 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:18:14.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:14.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:18:14.136 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:18:14.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:14.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:18:14.364 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:14.350+0000 7fad4e0e6f40 -1 Falling back to public interface 2021-07-26T07:18:14.496 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:15.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:15.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:15.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:18:15.499 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:18:15.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:15.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:18:15.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:16.899 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:16.900 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:18:16.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:18:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:16.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:18:17.014 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:16.998+0000 7fad4e0e6f40 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T07:18:17.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:18.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:18.334 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:18.334 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:18:18.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:18:18.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:18.336 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:18:18.691 INFO:tasks.workunit.client.0.smithi094.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/327068,v1:127.0.0.1:6835/327068] [v2:127.0.0.1:6836/327068,v1:127.0.0.1:6837/327068] exists,up 2d431205-93fd-4781-be34-fa43af3a6a46 2021-07-26T07:18:18.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:18:18.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:18:18.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:18:18.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2021-07-26T07:18:18.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:18:18.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:18:18.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T07:18:18.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:18:18.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2021-07-26T07:18:18.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:18:18.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:18:18.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:18:18.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:18:18.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-07-26T07:18:18.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-07-26T07:18:18.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:18:18.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:18:18.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:18:18.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:18:18.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:18:18.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:18:18.702 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:18:18.702 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:18:18.702 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:18:18.702 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:18:18.702 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:18:18.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:18:18.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:18:18.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:18:18.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:18:18.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:18:18.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:18:18.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:18:18.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:18:18.704 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:18:18.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:18:18.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:18:18.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2021-07-26T07:18:18.705 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:18:18.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b21c3f79-15a3-4722-8c67-b9d32a2ebe02 2021-07-26T07:18:18.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 b21c3f79-15a3-4722-8c67-b9d32a2ebe02' 2021-07-26T07:18:18.707 INFO:tasks.workunit.client.0.smithi094.stdout:add osd5 b21c3f79-15a3-4722-8c67-b9d32a2ebe02 2021-07-26T07:18:18.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:18:18.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6Yf5gx7HjKRAA7qzp8TUTTVcLk5FLK+FJHg== 2021-07-26T07:18:18.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6Yf5gx7HjKRAA7qzp8TUTTVcLk5FLK+FJHg=="}' 2021-07-26T07:18:18.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b21c3f79-15a3-4722-8c67-b9d32a2ebe02 -i td/osd-recovery-stats/5/new.json 2021-07-26T07:18:19.305 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T07:18:19.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2021-07-26T07:18:19.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC6Yf5gx7HjKRAA7qzp8TUTTVcLk5FLK+FJHg== --osd-uuid b21c3f79-15a3-4722-8c67-b9d32a2ebe02 2021-07-26T07:18:19.349 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:19.334+0000 7fc40f5e7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:19.353 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:19.338+0000 7fc40f5e7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:19.355 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:19.342+0000 7fc40f5e7f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:19.355 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:19.342+0000 7fc40f5e7f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:18:19.355 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:19.342+0000 7fc40f5e7f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:18:19.356 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:19.342+0000 7fc40f5e7f40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:18:19.475 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:19.462+0000 7fc40f5e7f40 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-07-26T07:18:25.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-07-26T07:18:25.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:18:25.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T07:18:25.878 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd5 key to auth repository 2021-07-26T07:18:25.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:18:26.359 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.5 2021-07-26T07:18:26.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T07:18:26.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:18:26.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:18:26.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:18:26.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:18:26.388 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:26.374+0000 7fb14e4daf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:26.390 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:26.378+0000 7fb14e4daf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:26.394 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:26.378+0000 7fb14e4daf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:18:26.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T07:18:26.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:18:26.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T07:18:26.717 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:18:26.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:18:26.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:18:26.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:26.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:18:26.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:26.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:18:26.928 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:26.914+0000 7fb14e4daf40 -1 Falling back to public interface 2021-07-26T07:18:27.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:28.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:28.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:28.083 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:18:28.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:18:28.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:28.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:18:28.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:29.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:29.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:29.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:18:29.458 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:18:29.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:29.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:18:29.557 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:18:29.542+0000 7fb14e4daf40 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T07:18:29.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:18:30.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:18:30.835 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:18:30.836 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:18:30.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:18:30.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:18:30.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:18:31.202 INFO:tasks.workunit.client.0.smithi094.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/328474,v1:127.0.0.1:6843/328474] [v2:127.0.0.1:6844/328474,v1:127.0.0.1:6845/328474] exists,up b21c3f79-15a3-4722-8c67-b9d32a2ebe02 2021-07-26T07:18:31.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:18:31.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:18:31.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:18:31.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2021-07-26T07:18:31.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T07:18:31.797 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T07:18:31.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:18:32.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2021-07-26T07:18:33.656 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 4 2021-07-26T07:18:33.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2021-07-26T07:18:33.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:18:33.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:18:33.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:18:33.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:18:33.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:18:33.673 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:18:33.674 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:18:33.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:18:33.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:18:33.675 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:18:33.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:18:33.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:18:33.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:18:33.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:18:33.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:18:34.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:18:34.407 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:18:34.407 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:18:34.408 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:18:34.408 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:18:34.408 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:18:34.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:18:34.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:34.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:18:34.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836496 2021-07-26T07:18:34.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836496 2021-07-26T07:18:34.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496' 2021-07-26T07:18:34.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:34.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:18:34.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672973 2021-07-26T07:18:34.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672973 2021-07-26T07:18:34.827 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973' 2021-07-26T07:18:34.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:34.828 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:18:34.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509451 2021-07-26T07:18:34.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509451 2021-07-26T07:18:34.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451' 2021-07-26T07:18:34.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:34.970 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:18:35.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345928 2021-07-26T07:18:35.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345928 2021-07-26T07:18:35.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451 3-85899345928' 2021-07-26T07:18:35.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:35.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:18:35.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182406 2021-07-26T07:18:35.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182406 2021-07-26T07:18:35.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451 3-85899345928 4-107374182406' 2021-07-26T07:18:35.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:18:35.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:18:35.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018883 2021-07-26T07:18:35.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018883 2021-07-26T07:18:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836496 1-42949672973 2-64424509451 3-85899345928 4-107374182406 5-128849018883' 2021-07-26T07:18:35.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:35.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836496 2021-07-26T07:18:35.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:35.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:18:35.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836496 2021-07-26T07:18:35.425 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:35.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836496 2021-07-26T07:18:35.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-07-26T07:18:35.426 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836496 2021-07-26T07:18:35.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:18:35.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836496 -lt 21474836496 2021-07-26T07:18:35.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:35.841 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672973 2021-07-26T07:18:35.841 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:35.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:18:35.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672973 2021-07-26T07:18:35.845 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:35.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672973 2021-07-26T07:18:35.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-07-26T07:18:35.847 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672973 2021-07-26T07:18:35.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:18:36.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672973 -lt 42949672973 2021-07-26T07:18:36.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:36.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509451 2021-07-26T07:18:36.227 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:36.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:18:36.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509451 2021-07-26T07:18:36.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:36.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509451 2021-07-26T07:18:36.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2021-07-26T07:18:36.233 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509451 2021-07-26T07:18:36.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:18:36.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509451 -lt 64424509451 2021-07-26T07:18:36.595 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:36.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345928 2021-07-26T07:18:36.596 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:36.598 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:18:36.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345928 2021-07-26T07:18:36.599 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:36.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345928 2021-07-26T07:18:36.601 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2021-07-26T07:18:36.602 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345928 2021-07-26T07:18:36.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:18:36.993 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345928 -lt 85899345928 2021-07-26T07:18:36.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:36.994 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182406 2021-07-26T07:18:36.995 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:36.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:18:36.997 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:36.997 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182406 2021-07-26T07:18:36.998 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182406 2021-07-26T07:18:36.998 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182406 2021-07-26T07:18:36.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-07-26T07:18:36.999 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:18:37.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182406 2021-07-26T07:18:37.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:18:37.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018883 2021-07-26T07:18:37.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:18:37.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:18:37.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018883 2021-07-26T07:18:37.365 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:18:37.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018883 2021-07-26T07:18:37.366 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-07-26T07:18:37.367 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018883 2021-07-26T07:18:37.367 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:18:37.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018884 -lt 128849018883 2021-07-26T07:18:37.795 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:18:37.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:37.796 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:38.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:18:38.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:18:38.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:18:38.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:18:38.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:18:38.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:18:38.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:18:38.304 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:18:38.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:18:38.677 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:18:38.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:18:38.678 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:18:39.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:18:39.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:18:39.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:18:39.129 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2021-07-26T07:18:39.130 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:39.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2021-07-26T07:18:39.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:39.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2021-07-26T07:18:39.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:39.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2021-07-26T07:18:39.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:39.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2021-07-26T07:18:39.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:39.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2021-07-26T07:18:39.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:39.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2021-07-26T07:18:40.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.009 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2021-07-26T07:18:40.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2021-07-26T07:18:40.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2021-07-26T07:18:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2021-07-26T07:18:40.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2021-07-26T07:18:40.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.642 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2021-07-26T07:18:40.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2021-07-26T07:18:40.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2021-07-26T07:18:40.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:40.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2021-07-26T07:18:41.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:41.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2021-07-26T07:18:41.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:41.263 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2021-07-26T07:18:41.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:41.406 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2021-07-26T07:18:41.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:41.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2021-07-26T07:18:41.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:41.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2021-07-26T07:18:41.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:41.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2021-07-26T07:18:42.044 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2021-07-26T07:18:42.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2021-07-26T07:18:42.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2021-07-26T07:18:42.429 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.430 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2021-07-26T07:18:42.538 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.539 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2021-07-26T07:18:42.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2021-07-26T07:18:42.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2021-07-26T07:18:42.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:42.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2021-07-26T07:18:43.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:43.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2021-07-26T07:18:43.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:43.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2021-07-26T07:18:43.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:43.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2021-07-26T07:18:43.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:43.521 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2021-07-26T07:18:43.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:43.663 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2021-07-26T07:18:43.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:43.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2021-07-26T07:18:43.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:43.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2021-07-26T07:18:44.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.058 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2021-07-26T07:18:44.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2021-07-26T07:18:44.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2021-07-26T07:18:44.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.409 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2021-07-26T07:18:44.550 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2021-07-26T07:18:44.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2021-07-26T07:18:44.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2021-07-26T07:18:44.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:44.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2021-07-26T07:18:45.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:45.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2021-07-26T07:18:45.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:45.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2021-07-26T07:18:45.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:45.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2021-07-26T07:18:45.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:45.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2021-07-26T07:18:45.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:45.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2021-07-26T07:18:45.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:45.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2021-07-26T07:18:45.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:45.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2021-07-26T07:18:46.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:46.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2021-07-26T07:18:46.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:46.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2021-07-26T07:18:46.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:46.349 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2021-07-26T07:18:46.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:46.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2021-07-26T07:18:46.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:46.641 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2021-07-26T07:18:46.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:46.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2021-07-26T07:18:46.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:46.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2021-07-26T07:18:47.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:47.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2021-07-26T07:18:47.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:47.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2021-07-26T07:18:47.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:47.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2021-07-26T07:18:47.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:47.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2021-07-26T07:18:47.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:47.634 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2021-07-26T07:18:47.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:47.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2021-07-26T07:18:47.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:47.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2021-07-26T07:18:48.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2021-07-26T07:18:48.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2021-07-26T07:18:48.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2021-07-26T07:18:48.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2021-07-26T07:18:48.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2021-07-26T07:18:48.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2021-07-26T07:18:48.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2021-07-26T07:18:48.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:48.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2021-07-26T07:18:49.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:49.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2021-07-26T07:18:49.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:49.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2021-07-26T07:18:49.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:49.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2021-07-26T07:18:49.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:49.572 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2021-07-26T07:18:49.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:49.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2021-07-26T07:18:49.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:49.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2021-07-26T07:18:50.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.061 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2021-07-26T07:18:50.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2021-07-26T07:18:50.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2021-07-26T07:18:50.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2021-07-26T07:18:50.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2021-07-26T07:18:50.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.729 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2021-07-26T07:18:50.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2021-07-26T07:18:50.972 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:50.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2021-07-26T07:18:51.122 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:51.123 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2021-07-26T07:18:51.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:51.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2021-07-26T07:18:51.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:51.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2021-07-26T07:18:51.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:51.617 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2021-07-26T07:18:51.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:51.735 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2021-07-26T07:18:51.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:51.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2021-07-26T07:18:52.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.062 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2021-07-26T07:18:52.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2021-07-26T07:18:52.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2021-07-26T07:18:52.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2021-07-26T07:18:52.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.581 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2021-07-26T07:18:52.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2021-07-26T07:18:52.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2021-07-26T07:18:52.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:52.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2021-07-26T07:18:53.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:53.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2021-07-26T07:18:53.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:53.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2021-07-26T07:18:53.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:53.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2021-07-26T07:18:53.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:53.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2021-07-26T07:18:53.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:53.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2021-07-26T07:18:53.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:53.797 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2021-07-26T07:18:53.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:53.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2021-07-26T07:18:54.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:54.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2021-07-26T07:18:54.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:54.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2021-07-26T07:18:54.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:54.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2021-07-26T07:18:54.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:54.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2021-07-26T07:18:54.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:54.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2021-07-26T07:18:54.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:54.746 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2021-07-26T07:18:54.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:54.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2021-07-26T07:18:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:55.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2021-07-26T07:18:55.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:55.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2021-07-26T07:18:55.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:55.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2021-07-26T07:18:55.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:55.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2021-07-26T07:18:55.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:55.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2021-07-26T07:18:55.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:55.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2021-07-26T07:18:55.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:55.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2021-07-26T07:18:56.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:56.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2021-07-26T07:18:56.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:56.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2021-07-26T07:18:56.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:56.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2021-07-26T07:18:56.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:56.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2021-07-26T07:18:56.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:56.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2021-07-26T07:18:56.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:56.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2021-07-26T07:18:56.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:56.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2021-07-26T07:18:57.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:57.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2021-07-26T07:18:57.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:57.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2021-07-26T07:18:57.416 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:57.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2021-07-26T07:18:57.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:57.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2021-07-26T07:18:57.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:57.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2021-07-26T07:18:57.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:57.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2021-07-26T07:18:58.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2021-07-26T07:18:58.186 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2021-07-26T07:18:58.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2021-07-26T07:18:58.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2021-07-26T07:18:58.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2021-07-26T07:18:58.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2021-07-26T07:18:58.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2021-07-26T07:18:58.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:58.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2021-07-26T07:18:59.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:59.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2021-07-26T07:18:59.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:59.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2021-07-26T07:18:59.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:59.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2021-07-26T07:18:59.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:59.633 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2021-07-26T07:18:59.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:59.750 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2021-07-26T07:18:59.900 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:18:59.901 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2021-07-26T07:19:00.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2021-07-26T07:19:00.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2021-07-26T07:19:00.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2021-07-26T07:19:00.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.445 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2021-07-26T07:19:00.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2021-07-26T07:19:00.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.749 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2021-07-26T07:19:00.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2021-07-26T07:19:00.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:00.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2021-07-26T07:19:01.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:01.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2021-07-26T07:19:01.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:01.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2021-07-26T07:19:01.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:01.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2021-07-26T07:19:01.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:01.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2021-07-26T07:19:01.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:01.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2021-07-26T07:19:01.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:01.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2021-07-26T07:19:02.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2021-07-26T07:19:02.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2021-07-26T07:19:02.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2021-07-26T07:19:02.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.424 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2021-07-26T07:19:02.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2021-07-26T07:19:02.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2021-07-26T07:19:02.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.826 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2021-07-26T07:19:02.951 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:02.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2021-07-26T07:19:03.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:03.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2021-07-26T07:19:03.202 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:03.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2021-07-26T07:19:03.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:03.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2021-07-26T07:19:03.504 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:03.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2021-07-26T07:19:03.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:03.639 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2021-07-26T07:19:03.781 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:03.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2021-07-26T07:19:03.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:03.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2021-07-26T07:19:04.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:04.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2021-07-26T07:19:04.208 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:04.209 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2021-07-26T07:19:04.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:04.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2021-07-26T07:19:04.501 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:04.502 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2021-07-26T07:19:04.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:04.644 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2021-07-26T07:19:04.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:04.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2021-07-26T07:19:04.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:04.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2021-07-26T07:19:05.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:05.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2021-07-26T07:19:05.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:05.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2021-07-26T07:19:05.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:05.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2021-07-26T07:19:05.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:05.431 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2021-07-26T07:19:05.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:05.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2021-07-26T07:19:05.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:05.745 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2021-07-26T07:19:05.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:05.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2021-07-26T07:19:06.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2021-07-26T07:19:06.168 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2021-07-26T07:19:06.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2021-07-26T07:19:06.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.428 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2021-07-26T07:19:06.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2021-07-26T07:19:06.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2021-07-26T07:19:06.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2021-07-26T07:19:06.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-07-26T07:19:06.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2021-07-26T07:19:07.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2021-07-26T07:19:07.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:19:07.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:19:07.088 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:19:07.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:19:07.456 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2021-07-26T07:19:07.457 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2021-07-26T07:19:07.458 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T07:19:07.458 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:19:07.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test obj1 2021-07-26T07:19:07.459 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:19:07.808 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2021-07-26T07:19:07.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2021-07-26T07:19:07.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local poolname=test 2021-07-26T07:19:07.810 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: local objectname=obj1 2021-07-26T07:19:07.810 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: ceph --format json osd map test obj1 2021-07-26T07:19:07.811 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: jq '.acting | .[]' 2021-07-26T07:19:08.183 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:967: get_osds: local 'osds=1 2021-07-26T07:19:08.183 INFO:tasks.workunit.client.0.smithi094.stderr:0 2021-07-26T07:19:08.183 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:19:08.183 INFO:tasks.workunit.client.0.smithi094.stderr:4' 2021-07-26T07:19:08.184 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: get_osds: echo 1 0 2 4 2021-07-26T07:19:08.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2021-07-26T07:19:08.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2021-07-26T07:19:08.910 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is set 2021-07-26T07:19:08.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-07-26T07:19:08.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2021-07-26T07:19:09.636 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already out. 2021-07-26T07:19:09.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-07-26T07:19:09.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2021-07-26T07:19:10.171 INFO:tasks.workunit.client.0.smithi094.stderr:osd.0 is already out. 2021-07-26T07:19:10.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-07-26T07:19:10.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2021-07-26T07:19:10.967 INFO:tasks.workunit.client.0.smithi094.stderr:osd.2 is already out. 2021-07-26T07:19:10.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-07-26T07:19:10.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2021-07-26T07:19:12.020 INFO:tasks.workunit.client.0.smithi094.stderr:osd.4 is already out. 2021-07-26T07:19:12.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2021-07-26T07:19:13.295 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 2 2021-07-26T07:19:13.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2021-07-26T07:19:14.453 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is unset 2021-07-26T07:19:14.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2021-07-26T07:19:14.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:19:14.467 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:19:14.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:19:14.468 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:19:15.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2021-07-26T07:19:15.165 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T07:19:15.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2021-07-26T07:19:17.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2021-07-26T07:19:17.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:19:17.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:19:17.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:19:17.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:19:17.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:19:17.176 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:19:17.177 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:19:17.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:19:17.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:19:17.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:19:17.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:19:17.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:19:17.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:19:17.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:19:17.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:19:17.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:19:17.764 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:19:17.765 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:19:17.765 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:19:17.765 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:19:17.765 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:19:17.765 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:19:17.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:17.767 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:19:17.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836506 2021-07-26T07:19:17.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836506 2021-07-26T07:19:17.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506' 2021-07-26T07:19:17.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:17.915 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:19:18.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672984 2021-07-26T07:19:18.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672984 2021-07-26T07:19:18.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672984' 2021-07-26T07:19:18.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:18.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:19:18.275 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509461 2021-07-26T07:19:18.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509461 2021-07-26T07:19:18.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672984 2-64424509461' 2021-07-26T07:19:18.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:18.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:19:18.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345939 2021-07-26T07:19:18.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345939 2021-07-26T07:19:18.421 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672984 2-64424509461 3-85899345939' 2021-07-26T07:19:18.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:18.422 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:19:18.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182417 2021-07-26T07:19:18.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182417 2021-07-26T07:19:18.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672984 2-64424509461 3-85899345939 4-107374182417' 2021-07-26T07:19:18.571 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:19:18.572 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:19:18.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018894 2021-07-26T07:19:18.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018894 2021-07-26T07:19:18.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836506 1-42949672984 2-64424509461 3-85899345939 4-107374182417 5-128849018894' 2021-07-26T07:19:18.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:18.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836506 2021-07-26T07:19:18.726 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:18.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:19:18.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836506 2021-07-26T07:19:18.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:18.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836506 2021-07-26T07:19:18.729 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836506 2021-07-26T07:19:18.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2021-07-26T07:19:18.730 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:19.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836504 -lt 21474836506 2021-07-26T07:19:19.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:19:20.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:19:20.107 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:19:20.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836507 -lt 21474836506 2021-07-26T07:19:20.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:20.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672984 2021-07-26T07:19:20.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:20.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:19:20.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672984 2021-07-26T07:19:20.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:20.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672984 2021-07-26T07:19:20.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2021-07-26T07:19:20.481 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672984 2021-07-26T07:19:20.482 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:19:20.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672984 -lt 42949672984 2021-07-26T07:19:20.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:20.843 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509461 2021-07-26T07:19:20.844 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:20.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:19:20.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:20.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509461 2021-07-26T07:19:20.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509461 2021-07-26T07:19:20.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2021-07-26T07:19:20.848 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509461 2021-07-26T07:19:20.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:19:21.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509462 -lt 64424509461 2021-07-26T07:19:21.232 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:21.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345939 2021-07-26T07:19:21.233 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:21.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:19:21.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345939 2021-07-26T07:19:21.237 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:21.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345939 2021-07-26T07:19:21.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2021-07-26T07:19:21.239 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345939 2021-07-26T07:19:21.239 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:19:21.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345940 -lt 85899345939 2021-07-26T07:19:21.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:21.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182417 2021-07-26T07:19:21.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:21.662 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:19:21.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182417 2021-07-26T07:19:21.663 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:21.665 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182417 2021-07-26T07:19:21.665 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182417 2021-07-26T07:19:21.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2021-07-26T07:19:21.666 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:19:22.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182417 -lt 107374182417 2021-07-26T07:19:22.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:19:22.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018894 2021-07-26T07:19:22.029 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:19:22.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:19:22.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018894 2021-07-26T07:19:22.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:19:22.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018894 2021-07-26T07:19:22.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2021-07-26T07:19:22.035 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018894 2021-07-26T07:19:22.036 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:19:22.567 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018895 -lt 128849018894 2021-07-26T07:19:22.568 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:19:22.569 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:22.570 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:23.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:19:23.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:23.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:23.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:23.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:23.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:23.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:23.080 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:23.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:23.432 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:23.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:23.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:24.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:24.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:19:24.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:19:24.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:19:24.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:24.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:24.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:24.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:24.119 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:24.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:24.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:24.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:24.120 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:24.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:24.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:24.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:24.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:24.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:25.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:25.002 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:25.003 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:25.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:25.005 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:25.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:25.006 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:25.006 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:25.007 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:25.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:25.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:25.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:25.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:25.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:25.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:25.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:25.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:25.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:25.576 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:25.577 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:25.577 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:25.577 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:25.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:25.920 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:25.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:25.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:26.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:26.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:26.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:26.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:26.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:26.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:26.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:26.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:26.413 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:26.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:19:26.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:19:26.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:26.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:26.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:27.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:27.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:27.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:27.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:27.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:27.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:27.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:27.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:27.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:27.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:27.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:27.374 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:27.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:27.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:27.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:27.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:27.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:27.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:27.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:27.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:27.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:28.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:19:28.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:19:28.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:28.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:28.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:28.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:28.520 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:28.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:28.521 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:28.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:28.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:28.522 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:28.523 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:28.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:28.848 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:28.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:28.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:29.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:29.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:29.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:29.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:29.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:29.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:29.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:29.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:29.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:29.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:29.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:29.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:29.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:29.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:29.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:29.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:29.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:29.907 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:29.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:29.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:29.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:29.908 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:30.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:30.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:30.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:30.294 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:30.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:30.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:30.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:30.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:30.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:30.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:30.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:30.815 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:30.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:31.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:31.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:31.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:31.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:31.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:31.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:31.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:31.369 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:31.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:31.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:31.370 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:31.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:31.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:31.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:31.718 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:32.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:32.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:32.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:32.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:32.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:32.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:32.184 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:32.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:32.185 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:32.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:19:32.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:19:32.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:32.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:32.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:32.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:32.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:32.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:32.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:32.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:32.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:32.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:32.714 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:33.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:33.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:33.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:33.053 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:33.575 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:33.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:33.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:33.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:33.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:33.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:33.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:33.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:33.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:34.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:34.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:34.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:34.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:34.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:34.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:34.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:34.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:34.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:34.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:34.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:34.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:34.122 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:34.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:34.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:34.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:34.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:34.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:34.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:34.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:34.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:34.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:34.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:34.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:34.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:34.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:35.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:35.518 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:35.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:35.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:35.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:35.619 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:35.620 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:35.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:35.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:35.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:35.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:35.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:35.623 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:36.018 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:36.019 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:36.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:36.020 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:36.497 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:36.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:36.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:36.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:36.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:36.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:36.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:36.499 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:36.500 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:37.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:37.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:37.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:37.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:37.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:37.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:37.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:37.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:37.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:37.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:37.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:37.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:37.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:37.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:37.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:37.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:37.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:38.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:38.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:38.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:38.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:38.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:38.040 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:38.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:38.040 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:38.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:19:38.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:19:38.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:38.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:38.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:38.582 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:38.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:38.583 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:38.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:38.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:38.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:38.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:38.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:38.935 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:38.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:38.936 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:38.937 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:39.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:39.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:39.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:39.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:39.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:39.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:39.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:39.384 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:39.385 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:39.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:39.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:39.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:39.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:39.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:39.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:39.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:39.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:39.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:39.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:39.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:39.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:39.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:40.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:40.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:40.328 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:40.329 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:40.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:40.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:40.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:40.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:40.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:40.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:41.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:41.259 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:41.260 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:41.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:41.261 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:41.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:41.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:41.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:41.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:41.361 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:41.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:41.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:41.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:41.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:41.805 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:41.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:41.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:42.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:42.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:42.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:42.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:42.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:42.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:42.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:42.219 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:42.220 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:42.658 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:19:42.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:19:42.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:42.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:42.659 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:42.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:42.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:42.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:42.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:42.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:42.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:42.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:42.762 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:43.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:43.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:43.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:43.079 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:43.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:43.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:43.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:43.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:43.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:43.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:43.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:43.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:43.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:44.069 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:44.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:44.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:44.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:44.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:44.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:44.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:44.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:44.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:44.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:44.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:44.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:44.173 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:44.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:44.574 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:44.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:44.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:45.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:45.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:45.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:45.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:45.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:45.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:45.102 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:45.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:45.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:45.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:45.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:45.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:45.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:45.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:45.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:45.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:45.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:45.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:45.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:45.673 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:45.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:45.674 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:46.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:46.060 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:46.060 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:46.061 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:46.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:46.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:46.532 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:46.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:46.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:46.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:46.533 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:46.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:46.534 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:47.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:47.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:47.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:47.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:47.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:47.128 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:47.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:47.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:47.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:47.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:47.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:47.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:47.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:47.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:47.460 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:47.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:47.461 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:47.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:47.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:47.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:47.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:47.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:47.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:47.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:47.939 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:47.940 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:48.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:19:48.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:19:48.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:48.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:48.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:48.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:48.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:48.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:48.544 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:48.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:48.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:48.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:48.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:48.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:48.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:48.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:48.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:49.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:49.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:49.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:49.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:49.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:49.441 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:49.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:49.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:49.445 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:49.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:49.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:49.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:49.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:49.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:50.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:50.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:50.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:50.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:50.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:50.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:50.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:50.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:50.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:50.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:50.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:50.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:50.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:50.973 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:50.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:50.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:50.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:50.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:50.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:50.976 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:50.976 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:51.498 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:51.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:51.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:51.499 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:51.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:51.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:51.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:51.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:51.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:51.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:51.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:51.602 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:51.603 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:51.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:51.929 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:51.930 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:51.930 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:52.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:52.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:52.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:52.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:52.398 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:52.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:52.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:52.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:52.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:52.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:19:52.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:19:52.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:52.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:52.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:52.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:52.959 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:52.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:52.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:52.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:52.960 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:52.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:52.961 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:53.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:53.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:53.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:53.298 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:53.754 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:53.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:53.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:53.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:53.755 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:53.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:53.756 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:53.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:53.756 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:54.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:54.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:54.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:54.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:54.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:54.407 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:54.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:54.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:54.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:54.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:54.409 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:54.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:54.410 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:54.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:54.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:54.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:54.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:55.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:55.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:55.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:55.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:55.244 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:55.245 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:55.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:55.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:55.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:55.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:55.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:56.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:56.008 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:56.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:56.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:56.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:56.009 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:56.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:56.010 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:56.359 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:56.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:56.360 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:56.362 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:56.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:56.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:56.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:56.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:56.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:56.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:56.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:56.888 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:56.889 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:57.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:19:57.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:19:57.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:57.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:57.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:57.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:57.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:57.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:57.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:57.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:57.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:57.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:57.473 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:57.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:57.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:57.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:57.866 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:58.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:58.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:58.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:58.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:58.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:58.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:58.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:58.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:58.292 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:19:58.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:19:58.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:19:58.952 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:19:58.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:19:58.953 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:19:59.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:19:59.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:19:59.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:19:59.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:19:59.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:19:59.055 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:19:59.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:19:59.056 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:19:59.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:19:59.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:19:59.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:19:59.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:19:59.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:19:59.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:19:59.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:19:59.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:19:59.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:19:59.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:19:59.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:19:59.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:19:59.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:20:00.350 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:20:00.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:20:00.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:20:00.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:20:00.352 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:20:00.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:20:00.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:00.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:00.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:00.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:00.453 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:00.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:00.454 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:00.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:20:00.806 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:00.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:00.807 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:01.368 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:20:01.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:20:01.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:20:01.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:20:01.369 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:20:01.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:20:01.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:20:01.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:20:01.371 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:20:01.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:20:01.999 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:20:02.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:20:02.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:20:02.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:20:02.100 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:20:02.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:02.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:02.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:02.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:02.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:02.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:02.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:02.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:20:02.451 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:02.451 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:02.452 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:02.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:20:02.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:20:02.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:20:02.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:20:02.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:20:02.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:20:02.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:20:02.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:20:02.972 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:20:03.457 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:20:03.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:20:03.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:20:03.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:20:03.458 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:20:03.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:20:03.559 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:03.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:03.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:03.560 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:03.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:03.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:03.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:03.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:20:03.903 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:03.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:03.904 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:04.370 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:20:04.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:20:04.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:20:04.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:20:04.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:20:04.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:20:04.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:20:04.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:20:04.373 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:20:04.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=5 2021-07-26T07:20:04.813 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 5 '!=' null 2021-07-26T07:20:04.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:20:04.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:20:04.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:20:04.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:20:04.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:04.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:04.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:04.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:04.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:04.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:04.919 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:05.258 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:20:05.258 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:05.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:05.259 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:05.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:20:05.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:20:05.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:20:05.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:20:05.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:20:05.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:20:05.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:20:05.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:20:05.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:20:06.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:20:06.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:20:06.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:20:06.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:20:06.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:20:06.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:20:06.335 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:06.336 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:06.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:06.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:06.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:06.337 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:06.338 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:06.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:20:06.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:06.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:06.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:07.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:20:07.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:20:07.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:20:07.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:20:07.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:20:07.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:20:07.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:20:07.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:20:07.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:20:07.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:20:07.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:20:07.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:20:07.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:20:07.592 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:20:07.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:20:07.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:20:07.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:20:07.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:20:07.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:20:07.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:20:07.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:20:07.695 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:20:08.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:20:08.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:20:08.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:20:08.047 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:20:08.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:20:08.515 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:20:08.516 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:20:08.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2021-07-26T07:20:08.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:20:08.516 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:20:08.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:20:08.517 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:20:08.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2021-07-26T07:20:08.858 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2021-07-26T07:20:08.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2021-07-26T07:20:08.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2021-07-26T07:20:08.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-07-26T07:20:08.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-07-26T07:20:08.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2021-07-26T07:20:08.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-07-26T07:20:08.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2021-07-26T07:20:08.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-07-26T07:20:08.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2021-07-26T07:20:08.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-07-26T07:20:08.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-07-26T07:20:08.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-07-26T07:20:08.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-07-26T07:20:08.864 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-07-26T07:20:08.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-07-26T07:20:08.865 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.865 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-07-26T07:20:08.865 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-07-26T07:20:08.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2021-07-26T07:20:08.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-07-26T07:20:08.885 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:20:08.885 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.887 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2021-07-26T07:20:08.887 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-07-26T07:20:08.887 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-07-26T07:20:08.887 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:20:08.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2021-07-26T07:20:08.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2021-07-26T07:20:08.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2021-07-26T07:20:08.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:20:08.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2021-07-26T07:20:08.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:20:08.894 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:20:08.895 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.895 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-07-26T07:20:08.895 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:20:08.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-07-26T07:20:08.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-07-26T07:20:08.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:20:08.912 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:20:08.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:20:08.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:20:08.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-07-26T07:20:08.914 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:20:08.914 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.915 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2021-07-26T07:20:08.915 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-07-26T07:20:08.915 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-07-26T07:20:08.916 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:20:08.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2021-07-26T07:20:08.918 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2021-07-26T07:20:08.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2021-07-26T07:20:08.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:20:08.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2021-07-26T07:20:08.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:20:08.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:20:08.921 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.922 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-07-26T07:20:08.922 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:20:08.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-07-26T07:20:08.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-07-26T07:20:08.925 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:20:08.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:20:08.926 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:20:08.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:20:08.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-07-26T07:20:08.927 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2021-07-26T07:20:08.928 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.928 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-07-26T07:20:08.949 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2021-07-26T07:20:08.950 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2021-07-26T07:20:08.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2021-07-26T07:20:08.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:20:08.957 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2021-07-26T07:20:08.957 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2021-07-26T07:20:08.957 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-07-26T07:20:08.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:20:08.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-07-26T07:20:08.958 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:20:08.958 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2021-07-26T07:20:08.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2021-07-26T07:20:08.959 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:20:08.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2021-07-26T07:20:08.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2021-07-26T07:20:08.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:20:08.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:20:08.969 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:20:08.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:20:08.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2021-07-26T07:20:08.970 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T07:20:08.971 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T07:20:09.468 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T07:20:09.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2021-07-26T07:20:09.477 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:09.478 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:09.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:09.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:09.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:10.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:10.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-07-26T07:20:10.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:20:10.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:20:10.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:20:10.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:10.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:10.801 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:10.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:10.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:10.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:10.818 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:20:10.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:20:10.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:20:10.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:20:10.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:20:10.823 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:20:10.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:10.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:20:10.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:20:10.824 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:10.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:20:10.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:20:10.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:20:10.846 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:20:10.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:10.847 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:10.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:20:10.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:20:10.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:20:10.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-07-26T07:20:10.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-07-26T07:20:10.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-07-26T07:20:10.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-07-26T07:20:10.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:20:10.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:20:10.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:20:10.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:20:10.852 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:20:10.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:20:10.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:20:10.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:20:10.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:20:10.855 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:20:10.855 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:20:10.856 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:20:10.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:20:10.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:20:10.857 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:20:10.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:10.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:20:10.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:20:10.860 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:20:10.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:20:10.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:20:10.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:20:10.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:20:10.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:10.865 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:10.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:20:10.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:20:10.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:20:10.867 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-07-26T07:20:10.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:20:10.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:10.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:10.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257957 2021-07-26T07:20:10.873 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:20:10.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:20:10.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:20:10.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-07-26T07:20:10.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2021-07-26T07:20:10.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2021-07-26T07:20:10.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2021-07-26T07:20:10.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-07-26T07:20:10.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:20:10.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:20:10.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:20:10.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-07-26T07:20:10.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-07-26T07:20:11.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:20:11.283 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:11.284 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:11.284 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:11.285 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:11.285 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:11.285 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:11.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:20:11.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:20:11.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:20:11.590 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:20:11.591 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:20:11.591 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:20:11.592 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:20:11.593 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:20:11.593 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:20:11.593 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:20:11.593 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:20:11.594 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:11.594 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:11.595 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:20:11.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:20:11.595 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get fsid 2021-07-26T07:20:11.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:20:11.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:20:11.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:20:11.686 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:20:11.688 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:20:11.688 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:20:11.688 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:20:11.689 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:20:11.689 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:20:11.689 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:11.689 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:11.689 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:20:11.690 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:20:11.690 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get mon_host 2021-07-26T07:20:11.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2021-07-26T07:20:11.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2021-07-26T07:20:11.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:20:11.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:20:11.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:20:11.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2021-07-26T07:20:11.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:20:12.670 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:20:12.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:12.672 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:12.679 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:12.679 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:12.680 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:12.680 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:12.681 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:20:12.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:20:12.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2021-07-26T07:20:12.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:20:12.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:20:12.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:20:12.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:20:12.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:20:12.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:20:12.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:12.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:12.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:12.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:20:12.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:20:12.707 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:12.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:12.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:20:12.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:20:12.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:20:12.708 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:12.709 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:12.709 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:12.709 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:12.709 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:12.709 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:12.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:12.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:12.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:12.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:12.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:20:12.713 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:20:12.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:12.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:12.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:12.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:20:12.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:20:12.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:20:12.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:20:12.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c6075e4-96db-49c2-9213-7c4bb58d151a 2021-07-26T07:20:12.716 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 7c6075e4-96db-49c2-9213-7c4bb58d151a 2021-07-26T07:20:12.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7c6075e4-96db-49c2-9213-7c4bb58d151a' 2021-07-26T07:20:12.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:20:12.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsYv5gdKGpKhAAtzY02uz/69WoGjIEx8vdzQ== 2021-07-26T07:20:12.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsYv5gdKGpKhAAtzY02uz/69WoGjIEx8vdzQ=="}' 2021-07-26T07:20:12.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c6075e4-96db-49c2-9213-7c4bb58d151a -i td/osd-recovery-stats/0/new.json 2021-07-26T07:20:13.071 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:20:13.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2021-07-26T07:20:13.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAsYv5gdKGpKhAAtzY02uz/69WoGjIEx8vdzQ== --osd-uuid 7c6075e4-96db-49c2-9213-7c4bb58d151a 2021-07-26T07:20:13.109 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:13.093+0000 7fc0e2c7df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:13.111 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:13.097+0000 7fc0e2c7df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:13.114 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:13.101+0000 7fc0e2c7df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:13.115 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:13.101+0000 7fc0e2c7df40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:20:13.115 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:13.101+0000 7fc0e2c7df40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:20:13.115 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:13.101+0000 7fc0e2c7df40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:20:13.195 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:13.181+0000 7fc0e2c7df40 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-07-26T07:20:18.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-07-26T07:20:18.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:20:18.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:20:18.888 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:20:18.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:19.393 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:20:19.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:20:19.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:20:19.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:20:19.394 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:19.395 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:20:19.423 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:19.409+0000 7f3b3489df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:19.425 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:19.413+0000 7f3b3489df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:19.437 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:19.413+0000 7f3b3489df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:20:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:19.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:20:19.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:19.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:19.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:19.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:19.743 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:20:19.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:19.744 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:19.968 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:19.953+0000 7f3b3489df40 -1 Falling back to public interface 2021-07-26T07:20:20.106 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:21.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:21.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:21.109 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:20:21.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:21.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:21.110 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:21.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:22.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:22.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:22.463 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:20:22.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:20:22.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:22.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:22.570 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:22.557+0000 7f3b3489df40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:20:22.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:23.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:23.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:23.877 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:20:23.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:20:23.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:23.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:20:24.286 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/338166,v1:127.0.0.1:6803/338166] [v2:127.0.0.1:6804/338166,v1:127.0.0.1:6805/338166] exists,up 7c6075e4-96db-49c2-9213-7c4bb58d151a 2021-07-26T07:20:24.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:24.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:24.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:24.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2021-07-26T07:20:24.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:20:24.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:20:24.292 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:20:24.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:20:24.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:20:24.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:20:24.293 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:24.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:24.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:24.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:20:24.294 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:20:24.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:24.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:24.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:20:24.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:20:24.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:20:24.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:24.296 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:24.296 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:24.297 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:24.297 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:24.297 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:24.297 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:24.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:24.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:24.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:24.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:20:24.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:20:24.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:24.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:24.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:24.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:20:24.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:20:24.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:20:24.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:20:24.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=358aff5f-e3b5-4422-a206-30ab6514f739 2021-07-26T07:20:24.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 358aff5f-e3b5-4422-a206-30ab6514f739' 2021-07-26T07:20:24.302 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 358aff5f-e3b5-4422-a206-30ab6514f739 2021-07-26T07:20:24.303 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:20:24.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4Yv5gde8YEhAA5G7CLFTynsn/5l8N2UfIyA== 2021-07-26T07:20:24.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4Yv5gde8YEhAA5G7CLFTynsn/5l8N2UfIyA=="}' 2021-07-26T07:20:24.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 358aff5f-e3b5-4422-a206-30ab6514f739 -i td/osd-recovery-stats/1/new.json 2021-07-26T07:20:24.869 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:20:24.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2021-07-26T07:20:24.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA4Yv5gde8YEhAA5G7CLFTynsn/5l8N2UfIyA== --osd-uuid 358aff5f-e3b5-4422-a206-30ab6514f739 2021-07-26T07:20:24.912 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:24.897+0000 7f2ef6c42f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:24.938 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:24.921+0000 7f2ef6c42f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:24.939 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:24.925+0000 7f2ef6c42f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:24.940 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:24.925+0000 7f2ef6c42f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:20:24.940 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:24.925+0000 7f2ef6c42f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:20:24.941 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:24.925+0000 7f2ef6c42f40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:20:25.036 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:25.021+0000 7f2ef6c42f40 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-07-26T07:20:30.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-07-26T07:20:30.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:20:30.906 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:20:30.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:20:30.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:31.386 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:20:31.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:20:31.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:20:31.387 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:20:31.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:20:31.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:31.418 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:31.405+0000 7f0248ecbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:31.420 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:31.405+0000 7f0248ecbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:31.421 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:31.405+0000 7f0248ecbf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:31.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:20:31.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:31.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:20:31.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:31.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:31.769 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:31.770 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:20:31.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:31.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:31.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:20:31.983 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:31.969+0000 7f0248ecbf40 -1 Falling back to public interface 2021-07-26T07:20:32.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:33.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:33.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:33.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:33.155 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:20:33.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:33.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:20:33.522 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:34.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:34.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:34.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:20:34.525 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:20:34.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:34.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:20:34.696 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:34.681+0000 7f0248ecbf40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:20:34.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:35.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:35.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:35.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:20:35.937 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:20:35.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:35.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:20:36.301 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/339572,v1:127.0.0.1:6811/339572] [v2:127.0.0.1:6812/339572,v1:127.0.0.1:6813/339572] exists,up 358aff5f-e3b5-4422-a206-30ab6514f739 2021-07-26T07:20:36.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:36.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:36.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:36.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2021-07-26T07:20:36.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:20:36.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:20:36.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:20:36.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:20:36.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2021-07-26T07:20:36.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:20:36.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:36.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:36.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:36.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-07-26T07:20:36.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-07-26T07:20:36.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:36.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:36.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:20:36.309 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:20:36.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:20:36.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:36.310 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:36.310 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:36.310 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:36.310 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:36.311 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:36.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:36.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:36.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:36.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:36.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:20:36.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:20:36.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:36.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:36.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:36.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:20:36.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:20:36.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2021-07-26T07:20:36.315 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:20:36.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ebc7546-ef71-4d36-a7cc-ee4e112ba69e 2021-07-26T07:20:36.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2ebc7546-ef71-4d36-a7cc-ee4e112ba69e' 2021-07-26T07:20:36.317 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 2ebc7546-ef71-4d36-a7cc-ee4e112ba69e 2021-07-26T07:20:36.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:20:36.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEYv5ghHL/EhAAC6cup5vlFaK5rxMEunoSHw== 2021-07-26T07:20:36.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEYv5ghHL/EhAAC6cup5vlFaK5rxMEunoSHw=="}' 2021-07-26T07:20:36.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ebc7546-ef71-4d36-a7cc-ee4e112ba69e -i td/osd-recovery-stats/2/new.json 2021-07-26T07:20:36.863 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:20:36.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2021-07-26T07:20:36.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBEYv5ghHL/EhAAC6cup5vlFaK5rxMEunoSHw== --osd-uuid 2ebc7546-ef71-4d36-a7cc-ee4e112ba69e 2021-07-26T07:20:36.905 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:36.889+0000 7f0920cfcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:36.934 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:36.921+0000 7f0920cfcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:36.936 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:36.921+0000 7f0920cfcf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:36.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:36.921+0000 7f0920cfcf40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:20:36.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:36.921+0000 7f0920cfcf40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:20:36.937 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:36.921+0000 7f0920cfcf40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:20:36.980 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:36.965+0000 7f0920cfcf40 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-07-26T07:20:42.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-07-26T07:20:42.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:20:42.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:20:42.885 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T07:20:42.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:43.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:20:43.387 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T07:20:43.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:20:43.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:43.388 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:20:43.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:20:43.418 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:43.405+0000 7f0f9847df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:43.420 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:43.405+0000 7f0f9847df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:43.421 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:43.409+0000 7f0f9847df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:43.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:20:43.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:43.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:20:43.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:43.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:43.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:43.706 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:20:43.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:43.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:43.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:43.963 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:43.949+0000 7f0f9847df40 -1 Falling back to public interface 2021-07-26T07:20:44.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:45.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:45.084 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:45.085 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:20:45.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:45.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:45.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:45.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:46.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:46.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:46.447 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:20:46.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:20:46.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:46.448 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:46.548 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:46.532+0000 7f0f9847df40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:20:46.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:47.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:47.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:47.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:20:47.882 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:20:47.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:20:47.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:48.235 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/340978,v1:127.0.0.1:6819/340978] [v2:127.0.0.1:6820/340978,v1:127.0.0.1:6821/340978] exists,up 2ebc7546-ef71-4d36-a7cc-ee4e112ba69e 2021-07-26T07:20:48.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:20:48.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:20:48.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:20:48.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2021-07-26T07:20:48.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:20:48.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:20:48.244 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-07-26T07:20:48.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:20:48.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2021-07-26T07:20:48.245 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:20:48.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:20:48.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:20:48.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:20:48.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-07-26T07:20:48.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-07-26T07:20:48.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:20:48.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:20:48.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:20:48.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:20:48.248 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:20:48.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:20:48.249 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:20:48.249 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:20:48.249 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:20:48.249 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:20:48.250 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:48.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:20:48.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:20:48.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:20:48.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:20:48.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:20:48.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:20:48.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:20:48.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:20:48.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:48.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:20:48.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:20:48.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2021-07-26T07:20:48.252 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:20:48.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f3e5e4f-4e12-426a-a72c-4e88a7fa4217 2021-07-26T07:20:48.255 INFO:tasks.workunit.client.0.smithi094.stdout:add osd3 5f3e5e4f-4e12-426a-a72c-4e88a7fa4217 2021-07-26T07:20:48.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5f3e5e4f-4e12-426a-a72c-4e88a7fa4217' 2021-07-26T07:20:48.256 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:20:48.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQYv5gbw04DxAAKRH3tsV02qvmRf/7wn0ujg== 2021-07-26T07:20:48.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQYv5gbw04DxAAKRH3tsV02qvmRf/7wn0ujg=="}' 2021-07-26T07:20:48.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f3e5e4f-4e12-426a-a72c-4e88a7fa4217 -i td/osd-recovery-stats/3/new.json 2021-07-26T07:20:48.812 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:20:48.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2021-07-26T07:20:48.820 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBQYv5gbw04DxAAKRH3tsV02qvmRf/7wn0ujg== --osd-uuid 5f3e5e4f-4e12-426a-a72c-4e88a7fa4217 2021-07-26T07:20:48.853 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:48.836+0000 7ff2d04cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:48.866 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:48.852+0000 7ff2d04cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:48.867 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:48.852+0000 7ff2d04cff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:48.868 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:48.852+0000 7ff2d04cff40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:20:48.868 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:48.852+0000 7ff2d04cff40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:20:48.868 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:48.852+0000 7ff2d04cff40 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-07-26T07:20:48.969 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:48.952+0000 7ff2d04cff40 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-07-26T07:20:54.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-07-26T07:20:54.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:20:54.760 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd3 key to auth repository 2021-07-26T07:20:54.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2021-07-26T07:20:54.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:20:55.282 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.3 2021-07-26T07:20:55.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2021-07-26T07:20:55.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:20:55.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:20:55.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:20:55.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:20:55.312 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:55.296+0000 7f109306df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:55.342 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:55.328+0000 7f109306df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:55.343 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:55.328+0000 7f109306df40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:20:55.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2021-07-26T07:20:55.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:20:55.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=3 2021-07-26T07:20:55.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:20:55.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:20:55.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:55.694 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:20:55.694 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:20:55.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:55.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:20:55.899 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:55.884+0000 7f109306df40 -1 Falling back to public interface 2021-07-26T07:20:56.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:57.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:57.014 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:57.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:20:57.015 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:20:57.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:57.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:20:57.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:58.035 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:20:58.020+0000 7f109306df40 -1 osd.3 0 log_to_monitors {default=true} 2021-07-26T07:20:58.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:58.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:58.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:20:58.377 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:20:58.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:58.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:20:58.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:20:59.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:20:59.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:20:59.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:20:59.779 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:20:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:20:59.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.3 up' 2021-07-26T07:21:00.145 INFO:tasks.workunit.client.0.smithi094.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/342384,v1:127.0.0.1:6827/342384] [v2:127.0.0.1:6828/342384,v1:127.0.0.1:6829/342384] exists,up 5f3e5e4f-4e12-426a-a72c-4e88a7fa4217 2021-07-26T07:21:00.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:00.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:00.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:00.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2021-07-26T07:21:00.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:21:00.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:21:00.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-07-26T07:21:00.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:21:00.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2021-07-26T07:21:00.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:21:00.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:21:00.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:00.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:00.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-07-26T07:21:00.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-07-26T07:21:00.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:00.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:00.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:21:00.154 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:21:00.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:21:00.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:00.155 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:00.155 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:00.155 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:00.155 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:21:00.156 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:21:00.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:21:00.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:00.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:21:00.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:21:00.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:21:00.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:21:00.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:00.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:00.157 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:00.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:21:00.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:21:00.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2021-07-26T07:21:00.158 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:21:00.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f283804c-a823-499f-b401-cbc0f3194ac3 2021-07-26T07:21:00.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 f283804c-a823-499f-b401-cbc0f3194ac3' 2021-07-26T07:21:00.159 INFO:tasks.workunit.client.0.smithi094.stdout:add osd4 f283804c-a823-499f-b401-cbc0f3194ac3 2021-07-26T07:21:00.160 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:21:00.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcYv5gjNSwCRAAnITA9eKUjraCEkdKHcZ55A== 2021-07-26T07:21:00.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcYv5gjNSwCRAAnITA9eKUjraCEkdKHcZ55A=="}' 2021-07-26T07:21:00.173 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f283804c-a823-499f-b401-cbc0f3194ac3 -i td/osd-recovery-stats/4/new.json 2021-07-26T07:21:00.820 INFO:tasks.workunit.client.0.smithi094.stdout:4 2021-07-26T07:21:00.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2021-07-26T07:21:00.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBcYv5gjNSwCRAAnITA9eKUjraCEkdKHcZ55A== --osd-uuid f283804c-a823-499f-b401-cbc0f3194ac3 2021-07-26T07:21:00.872 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:00.856+0000 7f5b6c03af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:00.932 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:00.916+0000 7f5b6c03af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:00.934 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:00.920+0000 7f5b6c03af40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:00.935 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:00.920+0000 7f5b6c03af40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:21:00.935 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:00.920+0000 7f5b6c03af40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:21:00.935 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:00.920+0000 7f5b6c03af40 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-07-26T07:21:01.021 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:01.008+0000 7f5b6c03af40 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-07-26T07:21:07.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-07-26T07:21:07.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:21:07.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2021-07-26T07:21:07.029 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd4 key to auth repository 2021-07-26T07:21:07.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:21:07.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2021-07-26T07:21:07.510 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.4 2021-07-26T07:21:07.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:07.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:21:07.511 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:21:07.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:21:07.539 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:07.524+0000 7fb5235cef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:07.542 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:07.528+0000 7fb5235cef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:07.544 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:07.528+0000 7fb5235cef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:07.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2021-07-26T07:21:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:07.915 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=4 2021-07-26T07:21:07.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:07.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:07.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:07.918 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:21:07.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:07.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:07.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:21:08.085 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:08.068+0000 7fb5235cef40 -1 Falling back to public interface 2021-07-26T07:21:08.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:09.298 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:09.299 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:09.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:09.303 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:21:09.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:09.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:21:09.673 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:10.666 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:10.652+0000 7fb5235cef40 -1 osd.4 0 log_to_monitors {default=true} 2021-07-26T07:21:10.675 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:10.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:10.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:10.676 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:21:10.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:10.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:21:11.135 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:12.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:12.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:12.137 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:21:12.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:12.138 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:12.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.4 up' 2021-07-26T07:21:12.467 INFO:tasks.workunit.client.0.smithi094.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/343790,v1:127.0.0.1:6835/343790] [v2:127.0.0.1:6836/343790,v1:127.0.0.1:6837/343790] exists,up f283804c-a823-499f-b401-cbc0f3194ac3 2021-07-26T07:21:12.470 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:12.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:12.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:12.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2021-07-26T07:21:12.471 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:21:12.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:21:12.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-07-26T07:21:12.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:21:12.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2021-07-26T07:21:12.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:21:12.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:21:12.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:21:12.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:21:12.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-07-26T07:21:12.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-07-26T07:21:12.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:21:12.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:21:12.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:21:12.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:21:12.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:21:12.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:21:12.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:21:12.475 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:21:12.476 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:21:12.476 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:21:12.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:21:12.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:21:12.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:21:12.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:21:12.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:21:12.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:21:12.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:21:12.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:21:12.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:21:12.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:12.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:21:12.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:21:12.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2021-07-26T07:21:12.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:21:12.480 INFO:tasks.workunit.client.0.smithi094.stdout:add osd5 af0f9624-2433-4bf8-8b0f-c13bd83c041a 2021-07-26T07:21:12.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af0f9624-2433-4bf8-8b0f-c13bd83c041a 2021-07-26T07:21:12.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 af0f9624-2433-4bf8-8b0f-c13bd83c041a' 2021-07-26T07:21:12.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:21:12.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBoYv5gbUTeHBAAUfax4QVNaGdRuKGtJbFVQg== 2021-07-26T07:21:12.494 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBoYv5gbUTeHBAAUfax4QVNaGdRuKGtJbFVQg=="}' 2021-07-26T07:21:12.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af0f9624-2433-4bf8-8b0f-c13bd83c041a -i td/osd-recovery-stats/5/new.json 2021-07-26T07:21:13.229 INFO:tasks.workunit.client.0.smithi094.stdout:5 2021-07-26T07:21:13.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2021-07-26T07:21:13.250 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBoYv5gbUTeHBAAUfax4QVNaGdRuKGtJbFVQg== --osd-uuid af0f9624-2433-4bf8-8b0f-c13bd83c041a 2021-07-26T07:21:13.282 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:13.268+0000 7f6084daaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:13.284 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:13.268+0000 7f6084daaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:13.291 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:13.272+0000 7f6084daaf40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:13.292 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:13.272+0000 7f6084daaf40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:21:13.292 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:13.272+0000 7f6084daaf40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:21:13.292 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:13.272+0000 7f6084daaf40 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-07-26T07:21:13.359 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:13.344+0000 7f6084daaf40 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-07-26T07:21:19.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-07-26T07:21:19.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:21:19.621 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd5 key to auth repository 2021-07-26T07:21:19.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2021-07-26T07:21:19.622 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:21:20.112 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.5 2021-07-26T07:21:20.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2021-07-26T07:21:20.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:21:20.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:21:20.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:21:20.114 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:21:20.140 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:20.124+0000 7fa0f0901f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:20.142 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:20.128+0000 7fa0f0901f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:20.144 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:20.128+0000 7fa0f0901f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:21:20.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2021-07-26T07:21:20.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:21:20.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=5 2021-07-26T07:21:20.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:21:20.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:21:20.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:20.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:21:20.489 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:21:20.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:20.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:21:20.691 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:20.676+0000 7fa0f0901f40 -1 Falling back to public interface 2021-07-26T07:21:20.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:21.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:21.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:21.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:21:21.882 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:21:21.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:21.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:21:22.236 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:23.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:23.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:23.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:21:23.239 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:21:23.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:23.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:21:23.313 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:21:23.300+0000 7fa0f0901f40 -1 osd.5 0 log_to_monitors {default=true} 2021-07-26T07:21:23.666 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:21:24.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:21:24.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:21:24.669 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:21:24.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:21:24.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:21:24.671 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.5 up' 2021-07-26T07:21:25.070 INFO:tasks.workunit.client.0.smithi094.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/345196,v1:127.0.0.1:6843/345196] [v2:127.0.0.1:6844/345196,v1:127.0.0.1:6845/345196] exists,up af0f9624-2433-4bf8-8b0f-c13bd83c041a 2021-07-26T07:21:25.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:21:25.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:21:25.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:21:25.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2021-07-26T07:21:25.072 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T07:21:25.599 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T07:21:25.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:21:26.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2021-07-26T07:21:27.496 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 2 2021-07-26T07:21:27.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2021-07-26T07:21:27.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:21:27.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:21:27.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:21:27.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:21:27.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:21:27.511 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:21:27.512 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:21:27.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:21:27.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:21:27.512 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:21:27.731 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:21:27.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:21:27.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:21:27.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:21:27.733 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:21:28.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:21:28.090 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:21:28.090 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:21:28.090 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:21:28.091 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:21:28.091 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:21:28.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:21:28.091 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:28.092 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:21:28.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836495 2021-07-26T07:21:28.285 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836495 2021-07-26T07:21:28.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495' 2021-07-26T07:21:28.286 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:28.286 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:21:28.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672974 2021-07-26T07:21:28.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672974 2021-07-26T07:21:28.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2021-07-26T07:21:28.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:28.439 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:21:28.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509452 2021-07-26T07:21:28.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509452 2021-07-26T07:21:28.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2021-07-26T07:21:28.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:28.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:21:28.726 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345928 2021-07-26T07:21:28.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345928 2021-07-26T07:21:28.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452 3-85899345928' 2021-07-26T07:21:28.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:28.727 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:21:28.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182406 2021-07-26T07:21:28.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182406 2021-07-26T07:21:28.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452 3-85899345928 4-107374182406' 2021-07-26T07:21:28.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:21:28.888 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:21:29.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018883 2021-07-26T07:21:29.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018883 2021-07-26T07:21:29.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452 3-85899345928 4-107374182406 5-128849018883' 2021-07-26T07:21:29.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:29.021 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836495 2021-07-26T07:21:29.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:29.023 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:21:29.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836495 2021-07-26T07:21:29.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:29.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836495 2021-07-26T07:21:29.026 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836495 2021-07-26T07:21:29.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-07-26T07:21:29.027 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:21:29.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836495 -lt 21474836495 2021-07-26T07:21:29.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:29.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672974 2021-07-26T07:21:29.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:29.389 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:21:29.390 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672974 2021-07-26T07:21:29.391 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:29.391 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672974 2021-07-26T07:21:29.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-07-26T07:21:29.392 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672974 2021-07-26T07:21:29.393 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:21:29.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672974 -lt 42949672974 2021-07-26T07:21:29.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:29.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509452 2021-07-26T07:21:29.786 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:29.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:21:29.788 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509452 2021-07-26T07:21:29.789 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:29.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509452 2021-07-26T07:21:29.791 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-07-26T07:21:29.791 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509452 2021-07-26T07:21:29.792 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:21:30.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509452 -lt 64424509452 2021-07-26T07:21:30.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:30.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345928 2021-07-26T07:21:30.154 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:30.156 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:21:30.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345928 2021-07-26T07:21:30.157 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:30.158 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345928 2021-07-26T07:21:30.159 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2021-07-26T07:21:30.159 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345928 2021-07-26T07:21:30.160 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:21:30.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345926 -lt 85899345928 2021-07-26T07:21:30.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:21:31.551 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:21:31.551 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:21:31.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345929 -lt 85899345928 2021-07-26T07:21:31.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:31.917 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:31.918 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182406 2021-07-26T07:21:31.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:21:31.920 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182406 2021-07-26T07:21:31.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:31.922 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182406 2021-07-26T07:21:31.923 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-07-26T07:21:31.923 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182406 2021-07-26T07:21:31.924 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:21:32.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182406 -lt 107374182406 2021-07-26T07:21:32.319 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:21:32.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018883 2021-07-26T07:21:32.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:21:32.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:21:32.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018883 2021-07-26T07:21:32.323 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:21:32.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018883 2021-07-26T07:21:32.325 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-07-26T07:21:32.326 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018883 2021-07-26T07:21:32.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:21:32.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018884 -lt 128849018883 2021-07-26T07:21:32.741 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:21:32.742 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:32.742 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:33.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:21:33.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:21:33.221 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:21:33.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:21:33.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:21:33.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:21:33.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:21:33.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:21:33.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:21:33.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:21:33.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:21:33.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:21:34.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:21:34.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:21:34.067 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:21:34.068 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2021-07-26T07:21:34.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2021-07-26T07:21:34.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.183 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2021-07-26T07:21:34.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.360 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2021-07-26T07:21:34.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2021-07-26T07:21:34.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2021-07-26T07:21:34.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2021-07-26T07:21:34.828 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.829 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2021-07-26T07:21:34.996 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:34.997 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2021-07-26T07:21:35.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2021-07-26T07:21:35.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2021-07-26T07:21:35.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.356 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2021-07-26T07:21:35.490 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.491 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2021-07-26T07:21:35.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2021-07-26T07:21:35.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2021-07-26T07:21:35.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2021-07-26T07:21:35.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:35.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2021-07-26T07:21:36.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:36.059 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2021-07-26T07:21:36.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:36.367 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2021-07-26T07:21:36.492 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:36.493 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2021-07-26T07:21:36.593 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:36.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2021-07-26T07:21:36.711 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:36.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2021-07-26T07:21:36.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:36.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2021-07-26T07:21:37.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2021-07-26T07:21:37.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.163 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2021-07-26T07:21:37.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2021-07-26T07:21:37.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2021-07-26T07:21:37.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.505 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2021-07-26T07:21:37.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2021-07-26T07:21:37.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2021-07-26T07:21:37.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2021-07-26T07:21:37.965 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:37.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2021-07-26T07:21:38.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2021-07-26T07:21:38.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2021-07-26T07:21:38.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2021-07-26T07:21:38.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.451 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2021-07-26T07:21:38.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2021-07-26T07:21:38.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2021-07-26T07:21:38.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2021-07-26T07:21:38.902 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:38.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2021-07-26T07:21:39.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.012 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2021-07-26T07:21:39.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2021-07-26T07:21:39.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2021-07-26T07:21:39.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2021-07-26T07:21:39.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.480 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2021-07-26T07:21:39.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2021-07-26T07:21:39.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2021-07-26T07:21:39.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2021-07-26T07:21:39.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:39.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2021-07-26T07:21:40.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.057 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2021-07-26T07:21:40.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.182 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2021-07-26T07:21:40.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2021-07-26T07:21:40.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2021-07-26T07:21:40.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.509 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2021-07-26T07:21:40.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2021-07-26T07:21:40.761 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.762 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2021-07-26T07:21:40.910 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:40.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2021-07-26T07:21:41.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2021-07-26T07:21:41.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2021-07-26T07:21:41.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2021-07-26T07:21:41.404 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.405 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2021-07-26T07:21:41.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.512 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2021-07-26T07:21:41.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2021-07-26T07:21:41.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.764 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2021-07-26T07:21:41.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:41.898 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2021-07-26T07:21:42.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2021-07-26T07:21:42.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2021-07-26T07:21:42.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2021-07-26T07:21:42.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2021-07-26T07:21:42.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.500 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2021-07-26T07:21:42.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.600 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2021-07-26T07:21:42.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.725 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2021-07-26T07:21:42.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.851 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2021-07-26T07:21:42.994 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:42.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2021-07-26T07:21:43.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2021-07-26T07:21:43.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2021-07-26T07:21:43.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.354 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2021-07-26T07:21:43.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2021-07-26T07:21:43.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2021-07-26T07:21:43.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2021-07-26T07:21:43.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2021-07-26T07:21:43.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:43.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2021-07-26T07:21:44.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2021-07-26T07:21:44.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2021-07-26T07:21:44.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2021-07-26T07:21:44.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.423 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2021-07-26T07:21:44.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2021-07-26T07:21:44.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.625 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2021-07-26T07:21:44.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2021-07-26T07:21:44.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:44.928 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2021-07-26T07:21:45.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.043 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2021-07-26T07:21:45.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2021-07-26T07:21:45.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.269 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2021-07-26T07:21:45.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2021-07-26T07:21:45.486 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2021-07-26T07:21:45.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.612 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2021-07-26T07:21:45.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.722 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2021-07-26T07:21:45.846 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2021-07-26T07:21:45.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:45.955 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2021-07-26T07:21:46.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.055 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2021-07-26T07:21:46.180 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.181 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2021-07-26T07:21:46.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.289 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2021-07-26T07:21:46.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2021-07-26T07:21:46.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2021-07-26T07:21:46.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2021-07-26T07:21:46.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.825 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2021-07-26T07:21:46.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:46.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2021-07-26T07:21:47.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2021-07-26T07:21:47.227 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2021-07-26T07:21:47.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.344 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2021-07-26T07:21:47.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2021-07-26T07:21:47.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2021-07-26T07:21:47.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.687 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2021-07-26T07:21:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2021-07-26T07:21:47.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:47.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2021-07-26T07:21:48.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2021-07-26T07:21:48.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.155 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2021-07-26T07:21:48.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2021-07-26T07:21:48.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2021-07-26T07:21:48.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2021-07-26T07:21:48.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2021-07-26T07:21:48.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2021-07-26T07:21:48.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:48.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2021-07-26T07:21:49.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.017 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2021-07-26T07:21:49.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2021-07-26T07:21:49.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.243 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2021-07-26T07:21:49.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2021-07-26T07:21:49.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.452 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2021-07-26T07:21:49.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2021-07-26T07:21:49.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.686 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2021-07-26T07:21:49.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.804 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2021-07-26T07:21:49.903 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:49.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2021-07-26T07:21:50.045 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2021-07-26T07:21:50.146 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2021-07-26T07:21:50.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2021-07-26T07:21:50.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2021-07-26T07:21:50.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2021-07-26T07:21:50.605 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.606 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2021-07-26T07:21:50.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.732 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2021-07-26T07:21:50.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:50.866 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2021-07-26T07:21:51.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2021-07-26T07:21:51.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2021-07-26T07:21:51.241 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.242 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2021-07-26T07:21:51.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2021-07-26T07:21:51.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.469 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2021-07-26T07:21:51.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.569 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2021-07-26T07:21:51.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.670 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2021-07-26T07:21:51.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2021-07-26T07:21:51.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:51.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2021-07-26T07:21:52.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.004 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2021-07-26T07:21:52.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2021-07-26T07:21:52.254 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.255 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2021-07-26T07:21:52.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.355 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2021-07-26T07:21:52.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2021-07-26T07:21:52.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.589 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2021-07-26T07:21:52.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.690 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2021-07-26T07:21:52.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:52.865 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2021-07-26T07:21:53.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2021-07-26T07:21:53.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2021-07-26T07:21:53.225 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.226 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2021-07-26T07:21:53.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2021-07-26T07:21:53.442 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2021-07-26T07:21:53.552 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.553 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2021-07-26T07:21:53.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2021-07-26T07:21:53.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.778 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2021-07-26T07:21:53.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2021-07-26T07:21:53.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:53.995 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2021-07-26T07:21:54.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2021-07-26T07:21:54.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.213 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2021-07-26T07:21:54.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2021-07-26T07:21:54.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2021-07-26T07:21:54.546 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.547 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2021-07-26T07:21:54.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.664 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2021-07-26T07:21:54.839 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.840 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2021-07-26T07:21:54.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:54.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2021-07-26T07:21:55.115 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2021-07-26T07:21:55.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2021-07-26T07:21:55.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2021-07-26T07:21:55.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2021-07-26T07:21:55.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2021-07-26T07:21:55.676 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.677 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2021-07-26T07:21:55.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2021-07-26T07:21:55.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:55.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2021-07-26T07:21:56.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.011 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2021-07-26T07:21:56.136 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.137 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2021-07-26T07:21:56.238 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.239 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2021-07-26T07:21:56.337 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.338 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2021-07-26T07:21:56.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2021-07-26T07:21:56.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.563 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2021-07-26T07:21:56.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2021-07-26T07:21:56.822 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:56.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2021-07-26T07:21:57.015 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2021-07-26T07:21:57.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.153 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2021-07-26T07:21:57.282 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.283 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2021-07-26T07:21:57.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2021-07-26T07:21:57.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.517 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2021-07-26T07:21:57.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.669 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2021-07-26T07:21:57.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2021-07-26T07:21:57.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2021-07-26T07:21:57.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:57.986 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2021-07-26T07:21:58.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-07-26T07:21:58.087 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2021-07-26T07:21:58.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2021-07-26T07:21:58.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:21:58.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:21:58.222 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:21:58.223 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:21:58.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2021-07-26T07:21:58.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2021-07-26T07:21:58.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: get_pg: local poolname=test 2021-07-26T07:21:58.588 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: get_pg: local objectname=obj1 2021-07-26T07:21:58.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: jq -r .pgid 2021-07-26T07:21:58.589 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: get_pg: ceph --format json osd map test obj1 2021-07-26T07:21:58.948 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2021-07-26T07:21:58.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2021-07-26T07:21:58.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: local poolname=test 2021-07-26T07:21:58.949 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: local objectname=obj1 2021-07-26T07:21:58.950 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: get_primary test obj1 2021-07-26T07:21:58.950 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:21:58.951 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:21:58.951 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:21:58.951 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:21:59.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_not_primary: local primary=1 2021-07-26T07:21:59.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-07-26T07:21:59.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_not_primary: ceph --format json osd map test obj1 2021-07-26T07:21:59.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2021-07-26T07:21:59.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2021-07-26T07:22:00.424 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is set 2021-07-26T07:22:00.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2021-07-26T07:22:01.100 INFO:tasks.workunit.client.0.smithi094.stderr:osd.1 is already out. osd.0 is already out. 2021-07-26T07:22:01.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2021-07-26T07:22:02.829 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 4 2021-07-26T07:22:02.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2021-07-26T07:22:04.088 INFO:tasks.workunit.client.0.smithi094.stderr:norecover is unset 2021-07-26T07:22:04.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2021-07-26T07:22:04.103 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: local poolname=test 2021-07-26T07:22:04.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: get_primary: local objectname=obj1 2021-07-26T07:22:04.104 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: get_primary: ceph --format json osd map test obj1 2021-07-26T07:22:04.105 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: get_primary: jq .acting_primary 2021-07-26T07:22:04.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2021-07-26T07:22:04.525 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2021-07-26T07:22:04.761 INFO:tasks.workunit.client.0.smithi094.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-07-26T07:22:04.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2021-07-26T07:22:06.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2021-07-26T07:22:06.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:22:06.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:22:06.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:22:06.771 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:22:06.772 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:22:06.772 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:22:06.773 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:22:06.773 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:22:06.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:22:06.774 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:22:06.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:22:06.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:22:06.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:22:06.982 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:22:06.984 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:22:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:22:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:22:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:2 2021-07-26T07:22:07.647 INFO:tasks.workunit.client.0.smithi094.stderr:3 2021-07-26T07:22:07.648 INFO:tasks.workunit.client.0.smithi094.stderr:4 2021-07-26T07:22:07.648 INFO:tasks.workunit.client.0.smithi094.stderr:5' 2021-07-26T07:22:07.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:22:07.648 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:22:07.648 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:22:07.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836505 2021-07-26T07:22:07.782 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836505 2021-07-26T07:22:07.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505' 2021-07-26T07:22:07.783 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:22:07.783 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:22:07.991 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672984 2021-07-26T07:22:07.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672984 2021-07-26T07:22:07.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672984' 2021-07-26T07:22:07.992 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:22:07.993 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:22:08.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509462 2021-07-26T07:22:08.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509462 2021-07-26T07:22:08.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509462' 2021-07-26T07:22:08.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:22:08.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-07-26T07:22:08.287 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=85899345938 2021-07-26T07:22:08.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 85899345938 2021-07-26T07:22:08.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509462 3-85899345938' 2021-07-26T07:22:08.288 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:22:08.288 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-07-26T07:22:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=107374182416 2021-07-26T07:22:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 107374182416 2021-07-26T07:22:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509462 3-85899345938 4-107374182416' 2021-07-26T07:22:08.436 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:22:08.437 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-07-26T07:22:08.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=128849018893 2021-07-26T07:22:08.573 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 128849018893 2021-07-26T07:22:08.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509462 3-85899345938 4-107374182416 5-128849018893' 2021-07-26T07:22:08.574 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:22:08.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836505 2021-07-26T07:22:08.575 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:22:08.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:22:08.577 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836505 2021-07-26T07:22:08.577 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:22:08.579 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836505 2021-07-26T07:22:08.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836505 2021-07-26T07:22:08.580 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2021-07-26T07:22:08.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:22:09.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836505 -lt 21474836505 2021-07-26T07:22:09.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:22:09.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672984 2021-07-26T07:22:09.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:22:09.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:22:09.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672984 2021-07-26T07:22:09.087 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:22:09.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672984 2021-07-26T07:22:09.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2021-07-26T07:22:09.089 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672984 2021-07-26T07:22:09.090 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:22:09.446 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672984 -lt 42949672984 2021-07-26T07:22:09.447 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:22:09.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509462 2021-07-26T07:22:09.448 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:22:09.450 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:22:09.451 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509462 2021-07-26T07:22:09.451 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:22:09.454 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509462 2021-07-26T07:22:09.454 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509462 2021-07-26T07:22:09.455 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2021-07-26T07:22:09.455 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:22:09.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509462 -lt 64424509462 2021-07-26T07:22:09.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:22:09.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 3-85899345938 2021-07-26T07:22:09.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:22:09.924 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=3 2021-07-26T07:22:09.925 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 3-85899345938 2021-07-26T07:22:09.926 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:22:09.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=85899345938 2021-07-26T07:22:09.927 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2021-07-26T07:22:09.928 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.3 seq 85899345938 2021-07-26T07:22:09.929 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 3 2021-07-26T07:22:10.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 85899345938 -lt 85899345938 2021-07-26T07:22:10.348 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:22:10.348 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 4-107374182416 2021-07-26T07:22:10.349 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:22:10.351 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=4 2021-07-26T07:22:10.352 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:22:10.352 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 4-107374182416 2021-07-26T07:22:10.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=107374182416 2021-07-26T07:22:10.353 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2021-07-26T07:22:10.354 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.4 seq 107374182416 2021-07-26T07:22:10.354 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 4 2021-07-26T07:22:10.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 107374182416 -lt 107374182416 2021-07-26T07:22:10.709 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:22:10.710 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 5-128849018893 2021-07-26T07:22:10.710 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:22:10.712 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=5 2021-07-26T07:22:10.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 5-128849018893 2021-07-26T07:22:10.713 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:22:10.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=128849018893 2021-07-26T07:22:10.716 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.5 seq 128849018893 2021-07-26T07:22:10.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2021-07-26T07:22:10.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 5 2021-07-26T07:22:11.129 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 128849018894 -lt 128849018893 2021-07-26T07:22:11.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:22:11.130 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:11.131 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:11.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:22:11.583 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:11.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:11.584 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:11.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:11.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:11.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:11.585 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:11.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:11.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:11.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:11.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:12.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:12.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' -1 2021-07-26T07:22:12.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: loop=0 2021-07-26T07:22:12.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: num_active_clean=0 2021-07-26T07:22:12.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:12.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:12.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:12.526 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:12.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:12.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:12.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:12.528 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:12.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:12.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:12.943 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:12.944 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:12.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:12.945 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:13.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:13.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:13.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:13.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:13.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:13.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:13.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:13.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:13.479 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:14.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=1 2021-07-26T07:22:14.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 1 '!=' null 2021-07-26T07:22:14.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:14.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:14.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:14.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:14.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:14.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:14.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:14.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:14.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:14.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:14.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:14.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:14.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:14.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:14.615 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:15.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:15.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:15.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:15.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:15.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:15.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:15.133 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:15.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:15.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:15.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:15.556 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:15.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:15.557 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:15.558 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:15.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:15.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:15.658 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:15.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:15.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:15.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:15.659 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:15.660 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:16.070 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:16.071 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:16.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:16.072 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:16.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:16.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:16.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:16.523 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:16.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:16.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:16.524 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:16.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:16.525 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:17.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:17.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:17.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:17.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:17.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:17.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:17.206 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:17.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:17.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:17.207 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:17.561 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:17.561 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:17.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:17.562 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:18.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:18.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:18.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:18.038 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:18.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:18.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:18.039 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:18.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:18.039 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:18.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:18.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:18.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:18.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:18.588 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:18.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:18.689 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:18.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:18.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:18.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:18.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:18.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:18.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:19.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:19.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:19.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:19.033 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:19.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:19.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:19.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:19.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:19.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:19.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:19.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:19.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:19.476 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:19.919 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:19.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:19.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:19.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:19.921 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:20.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:20.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:20.022 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:20.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:20.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:20.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:20.023 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:20.024 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:20.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:20.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:20.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:20.388 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:20.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:20.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:20.862 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:20.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:20.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:20.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:20.863 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:20.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:20.864 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:21.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:21.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:21.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:21.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:21.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:21.481 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:21.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:21.483 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:21.483 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:21.483 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:21.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:21.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:21.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:21.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:21.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:21.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:21.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:22.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:22.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:22.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:22.272 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:22.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:22.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:22.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:22.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:22.274 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:22.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:22.847 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:22.848 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:22.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:22.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:22.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:22.950 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:22.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:22.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:22.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:22.951 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:22.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:22.952 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:23.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:23.300 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:23.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:23.301 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:23.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:23.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:23.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:23.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:23.788 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:23.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:23.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:23.793 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:23.794 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:24.273 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:24.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:24.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:24.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:24.274 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:24.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:24.375 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:24.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:24.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:24.376 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:24.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:24.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:24.377 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:24.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:24.758 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:24.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:24.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:25.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:25.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:25.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:25.265 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:25.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:25.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:25.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:25.269 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:25.269 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:25.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:25.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:25.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:25.887 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:25.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:25.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:25.988 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:25.989 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:25.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:25.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:25.990 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:25.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:25.991 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:26.323 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:26.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:26.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:26.324 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:26.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:26.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:26.896 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:26.897 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:26.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:26.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:27.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:27.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:27.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:27.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:27.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:27.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:27.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:27.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:27.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:27.546 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:27.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:27.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:27.547 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:27.886 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:27.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:27.887 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:27.888 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:28.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:28.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:28.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:28.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:28.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:28.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:28.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:28.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:28.363 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:28.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:28.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:28.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:28.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:28.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:28.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:28.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:28.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:28.933 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:28.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:28.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:28.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:28.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:29.279 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:29.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:29.280 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:29.281 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:29.817 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:29.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:29.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:29.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:29.818 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:29.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:29.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:29.819 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:29.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:30.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:30.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:30.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:30.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:30.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:30.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:30.432 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:30.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:30.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:30.433 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:30.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:30.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:30.434 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:30.920 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:30.921 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:30.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:30.922 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:31.384 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:31.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:31.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:31.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:31.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:31.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:31.386 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:31.386 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:31.387 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:31.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:31.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:31.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:31.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:31.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:31.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:31.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:31.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:31.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:31.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:31.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:31.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:31.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:32.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:32.316 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:32.316 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:32.316 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:32.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:32.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:32.830 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:32.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:32.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:32.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:32.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:32.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:32.832 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:33.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:33.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:33.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:33.371 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:33.372 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:33.472 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:33.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:33.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:33.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:33.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:33.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:33.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:33.475 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:33.831 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:33.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:33.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:33.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:34.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:34.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:34.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:34.373 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:34.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:34.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:34.374 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:34.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:34.375 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:34.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:34.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:34.794 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:34.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:34.795 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:34.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:34.895 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:34.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:34.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:34.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:34.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:34.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:34.898 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:35.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:35.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:35.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:35.253 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:35.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:35.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:35.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:35.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:35.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:35.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:35.701 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:35.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:35.702 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:36.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:36.169 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:36.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:36.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:36.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:36.270 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:36.271 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:36.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:36.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:36.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:36.272 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:36.273 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:36.273 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:36.599 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:36.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:36.600 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:36.601 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:37.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:37.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:37.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:37.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:37.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:37.076 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:37.078 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:37.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:37.078 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:37.586 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:37.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:37.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:37.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:37.587 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:37.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:37.688 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:37.689 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:37.689 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:37.690 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:37.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:37.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:37.691 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:38.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:38.142 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:38.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:38.143 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:38.716 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:38.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:38.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:38.717 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:38.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:38.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:38.718 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:38.719 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:38.720 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:39.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:39.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:39.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:39.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:39.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:39.417 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:39.418 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:39.419 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:39.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:39.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:39.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:39.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:39.421 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:39.780 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:39.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:39.781 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:39.782 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:40.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:40.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:40.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:40.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:40.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:40.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:40.331 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:40.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:40.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:40.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:40.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:40.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:40.806 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:40.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:40.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:40.907 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:40.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:40.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:40.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:40.909 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:40.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:40.910 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:41.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:41.306 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:41.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:41.307 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:41.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:41.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:41.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:41.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:41.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:41.799 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:41.800 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:41.800 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:41.801 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:42.228 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:42.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:42.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:42.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:42.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:42.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:42.330 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:42.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:42.331 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:42.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:42.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:42.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:42.332 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:42.748 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:42.748 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:42.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:42.749 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:43.233 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:43.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:43.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:43.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:43.234 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:43.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:43.235 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:43.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:43.236 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:43.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:43.736 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:43.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:43.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:43.737 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:43.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:43.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:43.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:43.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:43.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:43.839 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:43.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:43.840 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:44.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:44.176 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:44.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:44.177 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:44.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:44.680 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:44.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:44.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:44.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:44.681 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:44.682 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:44.682 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:44.683 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:45.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:45.174 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:45.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:45.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:45.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:45.276 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:45.277 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:45.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:45.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:45.277 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:45.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:45.278 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:45.279 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:45.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:45.624 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:45.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:45.625 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:46.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:46.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:46.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:46.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:46.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:46.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:46.047 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:46.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:46.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:46.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:46.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:46.613 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:46.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:46.614 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:46.714 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:46.715 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:46.715 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:46.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:46.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:46.716 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:46.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:46.717 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:47.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:47.085 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:47.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:47.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:47.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:47.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:47.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:47.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:47.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:47.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:47.542 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:47.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:47.542 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:48.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:48.065 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:48.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:48.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:48.066 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:48.166 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:48.167 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:48.167 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:48.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:48.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:48.168 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:48.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:48.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:48.510 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:48.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:48.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:48.511 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:48.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:48.974 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:48.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:48.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:48.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:48.976 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:48.977 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:48.977 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:48.978 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:49.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:49.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:49.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:49.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:49.427 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:49.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:49.527 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:49.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:49.529 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:49.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:49.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:49.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:49.530 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:49.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:49.871 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:49.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:49.872 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:50.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:50.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:50.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:50.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:50.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:50.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:50.318 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:50.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:50.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:50.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:50.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:50.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:50.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:50.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:50.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:50.985 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:50.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:50.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:50.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:50.986 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:50.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:50.987 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:51.324 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:51.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:51.326 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:51.327 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:51.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:51.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:51.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:51.814 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:51.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:51.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:51.815 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:51.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:51.816 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:52.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:52.295 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:52.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:52.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:52.296 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:52.396 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:52.397 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:52.397 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:52.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:52.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:52.398 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:52.399 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:52.400 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:52.753 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:52.753 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:52.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:52.754 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:53.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:53.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:53.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:53.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:53.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:53.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:53.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:53.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:53.254 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:53.775 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=3 2021-07-26T07:22:53.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 3 '!=' null 2021-07-26T07:22:53.776 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:53.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:53.777 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:53.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:53.877 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:53.878 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:53.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:53.879 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:53.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:53.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:53.880 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:54.229 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:54.230 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:54.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:54.231 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:54.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:54.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:54.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:54.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:54.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:54.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:54.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:54.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:54.701 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:55.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:55.217 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:55.218 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:55.219 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:55.220 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:55.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:55.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:55.318 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:55.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:55.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:55.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:55.319 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:55.320 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:55.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:55.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:55.684 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:55.685 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:56.124 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:56.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:56.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:56.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:56.125 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:56.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:56.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:56.126 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:56.127 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:56.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:56.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:56.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:56.590 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:56.591 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:56.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:56.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:56.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:56.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:56.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:56.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:56.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:56.693 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:57.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:57.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:57.030 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:57.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:57.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:57.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:57.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:57.478 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:57.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:57.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:57.479 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:57.480 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:57.481 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:58.026 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:22:58.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:22:58.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:58.027 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:58.028 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:58.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:58.132 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:58.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:58.132 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:58.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:58.133 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:58.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:58.134 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:58.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:58.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:58.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:58.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:22:58.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:22:58.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:22:58.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:22:58.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:22:58.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:22:58.933 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:22:58.934 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:22:58.934 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:22:58.935 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:22:59.361 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:22:59.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:22:59.362 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:22:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:22:59.363 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:22:59.462 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:22:59.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:22:59.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:22:59.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:22:59.463 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:22:59.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:22:59.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:22:59.464 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:22:59.807 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:22:59.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:22:59.808 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:22:59.809 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:23:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:23:00.463 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:23:00.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:23:00.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:23:00.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:23:00.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:23:00.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:23:00.465 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:23:00.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:23:00.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:23:00.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:23:00.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:23:00.979 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:23:01.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:23:01.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:23:01.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:23:01.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:23:01.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:23:01.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:23:01.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:23:01.082 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:23:01.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:23:01.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:23:01.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:01.412 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:01.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:23:01.904 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:23:01.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:23:01.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:23:01.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:23:01.905 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:23:01.906 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:23:01.906 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:23:01.906 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:23:02.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=4 2021-07-26T07:23:02.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 4 '!=' null 2021-07-26T07:23:02.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:23:02.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:23:02.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:23:02.482 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:23:02.483 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:23:02.483 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:23:02.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:23:02.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:23:02.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:23:02.484 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:23:02.485 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:23:02.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:23:02.812 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:23:02.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:02.813 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:03.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:23:03.300 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:23:03.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:23:03.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:23:03.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:23:03.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:23:03.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:23:03.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:23:03.302 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:23:03.792 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:23:03.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:23:03.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:23:03.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:23:03.793 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:23:03.893 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:23:03.894 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:23:03.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:23:03.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:23:03.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:23:03.896 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:23:03.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:23:03.897 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:23:04.240 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=0 2021-07-26T07:23:04.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:23:04.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:04.241 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:04.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 0 = 1 2021-07-26T07:23:04.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: test 0 '!=' 0 2021-07-26T07:23:04.705 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: get_is_making_recovery_progress 2021-07-26T07:23:04.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local recovery_progress 2021-07-26T07:23:04.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-07-26T07:23:04.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-07-26T07:23:04.706 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1279: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-07-26T07:23:04.706 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: ceph --format json status 2021-07-26T07:23:04.707 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-07-26T07:23:05.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1281: get_is_making_recovery_progress: local progress=2 2021-07-26T07:23:05.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1282: get_is_making_recovery_progress: test 2 '!=' null 2021-07-26T07:23:05.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1605: wait_for_clean: loop=0 2021-07-26T07:23:05.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: eval 2021-07-26T07:23:05.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1612: wait_for_clean: sleep 0.1 2021-07-26T07:23:05.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1613: wait_for_clean: loop+=1 2021-07-26T07:23:05.262 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:23:05.263 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:23:05.263 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:23:05.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:23:05.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:23:05.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:23:05.264 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:23:05.621 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:23:05.621 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:23:05.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:05.622 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:06.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:23:06.111 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:23:06.112 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:23:06.112 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2021-07-26T07:23:06.113 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2021-07-26T07:23:06.113 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2021-07-26T07:23:06.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2021-07-26T07:23:06.116 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2021-07-26T07:23:06.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-07-26T07:23:06.117 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-07-26T07:23:06.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2021-07-26T07:23:06.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-07-26T07:23:06.118 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2021-07-26T07:23:06.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-07-26T07:23:06.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2021-07-26T07:23:06.119 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-07-26T07:23:06.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2021-07-26T07:23:06.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2021-07-26T07:23:06.120 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-07-26T07:23:06.121 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-07-26T07:23:06.121 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-07-26T07:23:06.122 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.122 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-07-26T07:23:06.123 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-07-26T07:23:06.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2021-07-26T07:23:06.152 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-07-26T07:23:06.153 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:23:06.154 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.154 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2021-07-26T07:23:06.154 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-07-26T07:23:06.157 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-07-26T07:23:06.157 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:23:06.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2021-07-26T07:23:06.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2021-07-26T07:23:06.160 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2021-07-26T07:23:06.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:23:06.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2021-07-26T07:23:06.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:23:06.162 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:23:06.162 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.163 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-07-26T07:23:06.163 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:23:06.170 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-07-26T07:23:06.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-07-26T07:23:06.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:23:06.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:23:06.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:23:06.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:23:06.172 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-07-26T07:23:06.172 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:23:06.173 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.174 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2021-07-26T07:23:06.174 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-07-26T07:23:06.174 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:23:06.175 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-07-26T07:23:06.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2021-07-26T07:23:06.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2021-07-26T07:23:06.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2021-07-26T07:23:06.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:23:06.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2021-07-26T07:23:06.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:23:06.179 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:23:06.179 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.179 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-07-26T07:23:06.179 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:23:06.189 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-07-26T07:23:06.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-07-26T07:23:06.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:23:06.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:23:06.190 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:23:06.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:23:06.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2021-07-26T07:23:06.191 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2021-07-26T07:23:06.192 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:23:06.193 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.193 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2021-07-26T07:23:06.193 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2021-07-26T07:23:06.193 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2021-07-26T07:23:06.194 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:23:06.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2021-07-26T07:23:06.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2021-07-26T07:23:06.198 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-07-26T07:23:06.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-07-26T07:23:06.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-07-26T07:23:06.199 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-07-26T07:23:06.200 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-07-26T07:23:06.200 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2021-07-26T07:23:06.201 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2021-07-26T07:23:06.201 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2021-07-26T07:23:06.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2021-07-26T07:23:06.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2021-07-26T07:23:06.214 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-07-26T07:23:06.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-07-26T07:23:06.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-07-26T07:23:06.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-07-26T07:23:06.215 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2021-07-26T07:23:06.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2021-07-26T07:23:06.216 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-07-26T07:23:06.699 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' does not exist 2021-07-26T07:23:06.708 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2021-07-26T07:23:06.709 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:23:06.709 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:23:06.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:23:06.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:23:06.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:23:08.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:23:08.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-07-26T07:23:08.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:23:08.029 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:23:08.030 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:23:08.031 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:23:08.032 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:23:08.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:23:08.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:23:08.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:23:08.046 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:23:08.046 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:23:08.048 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:23:08.048 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:23:08.049 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:23:08.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:23:08.050 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:23:08.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:08.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:23:08.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:23:08.052 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:08.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:23:08.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:23:08.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:23:08.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:23:08.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:08.077 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:08.077 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:23:08.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:23:08.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:23:08.079 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-07-26T07:23:08.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-07-26T07:23:08.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-07-26T07:23:08.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-07-26T07:23:08.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-07-26T07:23:08.080 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-07-26T07:23:08.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-07-26T07:23:08.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-07-26T07:23:08.081 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-07-26T07:23:08.081 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-07-26T07:23:08.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-07-26T07:23:08.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-07-26T07:23:08.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-07-26T07:23:08.083 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-07-26T07:23:08.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-07-26T07:23:08.084 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-07-26T07:23:08.085 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-07-26T07:23:08.086 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-07-26T07:23:08.086 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-07-26T07:23:08.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:08.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-07-26T07:23:08.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-07-26T07:23:08.089 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-07-26T07:23:08.090 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-07-26T07:23:08.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-07-26T07:23:08.092 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-07-26T07:23:08.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-07-26T07:23:08.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:08.094 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:08.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.257957 2021-07-26T07:23:08.095 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-07-26T07:23:08.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-07-26T07:23:08.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-07-26T07:23:08.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-07-26T07:23:08.098 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:08.099 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:08.099 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.257957 2021-07-26T07:23:08.101 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-07-26T07:23:08.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-07-26T07:23:08.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-07-26T07:23:08.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-07-26T07:23:08.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2021-07-26T07:23:08.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2021-07-26T07:23:08.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2021-07-26T07:23:08.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2021-07-26T07:23:08.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-07-26T07:23:08.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-07-26T07:23:08.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-07-26T07:23:08.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-07-26T07:23:08.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-07-26T07:23:08.105 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-07-26T07:23:08.471 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-07-26T07:23:08.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:08.472 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:08.473 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:08.473 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:08.474 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:08.474 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:08.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-07-26T07:23:08.757 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-07-26T07:23:08.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-07-26T07:23:08.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:23:08.759 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:23:08.760 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=fsid 2021-07-26T07:23:08.761 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .fsid 2021-07-26T07:23:08.761 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:23:08.762 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:23:08.762 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:23:08.762 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:23:08.763 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:08.763 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:08.764 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:23:08.764 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:23:08.765 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get fsid 2021-07-26T07:23:08.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-07-26T07:23:08.849 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: local daemon=mon 2021-07-26T07:23:08.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: get_config: local id=a 2021-07-26T07:23:08.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: local config=mon_host 2021-07-26T07:23:08.850 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: get_config: jq -r .mon_host 2021-07-26T07:23:08.852 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: get_asok_path mon.a 2021-07-26T07:23:08.852 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-07-26T07:23:08.852 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-07-26T07:23:08.853 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-07-26T07:23:08.853 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:08.853 INFO:tasks.workunit.client.0.smithi094.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:08.854 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.257957/ceph-mon.a.asok 2021-07-26T07:23:08.854 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: CEPH_ARGS= 2021-07-26T07:23:08.854 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: get_config: ceph --format json daemon /tmp/ceph-asok.257957/ceph-mon.a.asok config get mon_host 2021-07-26T07:23:08.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2021-07-26T07:23:08.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2021-07-26T07:23:08.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-07-26T07:23:08.938 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-07-26T07:23:08.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-07-26T07:23:08.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2021-07-26T07:23:08.939 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-07-26T07:23:09.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-07-26T07:23:09.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:09.831 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:09.832 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:09.832 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:09.833 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:09.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:09.833 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:23:09.834 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-07-26T07:23:09.865 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2021-07-26T07:23:09.866 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2021-07-26T07:23:09.869 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:23:09.870 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2021-07-26T07:23:09.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:23:09.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:23:09.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-07-26T07:23:09.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:23:09.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2021-07-26T07:23:09.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:23:09.872 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:23:09.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:23:09.873 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:23:09.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-07-26T07:23:09.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-07-26T07:23:09.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:23:09.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:23:09.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:23:09.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:23:09.875 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:23:09.875 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:09.876 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:09.876 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:09.876 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:09.876 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:09.877 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:09.879 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:09.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:23:09.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:23:09.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:23:09.881 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:23:09.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:23:09.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:23:09.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:23:09.882 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:09.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:23:09.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:23:09.883 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2021-07-26T07:23:09.884 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:23:09.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c0e2c69-a90e-42fa-b48d-442025ee10fd 2021-07-26T07:23:09.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8c0e2c69-a90e-42fa-b48d-442025ee10fd' 2021-07-26T07:23:09.885 INFO:tasks.workunit.client.0.smithi094.stdout:add osd0 8c0e2c69-a90e-42fa-b48d-442025ee10fd 2021-07-26T07:23:09.885 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:23:09.888 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdYv5g2EBkNBAAsSOUGF0mU3GCvy5KdxVPEA== 2021-07-26T07:23:09.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdYv5g2EBkNBAAsSOUGF0mU3GCvy5KdxVPEA=="}' 2021-07-26T07:23:09.889 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c0e2c69-a90e-42fa-b48d-442025ee10fd -i td/osd-recovery-stats/0/new.json 2021-07-26T07:23:10.235 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:23:10.247 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2021-07-26T07:23:10.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdYv5g2EBkNBAAsSOUGF0mU3GCvy5KdxVPEA== --osd-uuid 8c0e2c69-a90e-42fa-b48d-442025ee10fd 2021-07-26T07:23:10.272 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:10.263+0000 7f7d33d83f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:10.274 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:10.263+0000 7f7d33d83f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:10.275 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:10.267+0000 7f7d33d83f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:10.276 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:10.267+0000 7f7d33d83f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:23:10.276 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:10.267+0000 7f7d33d83f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:23:10.277 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:10.267+0000 7f7d33d83f40 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-07-26T07:23:10.365 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:10.355+0000 7f7d33d83f40 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-07-26T07:23:16.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-07-26T07:23:16.019 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:23:16.021 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd0 key to auth repository 2021-07-26T07:23:16.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2021-07-26T07:23:16.022 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:23:16.505 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.0 2021-07-26T07:23:16.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2021-07-26T07:23:16.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:16.507 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:23:16.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:23:16.508 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:23:16.534 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:16.523+0000 7f8282c1ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:16.536 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:16.527+0000 7f8282c1ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:16.538 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:16.527+0000 7f8282c1ef40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:16.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2021-07-26T07:23:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:23:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=0 2021-07-26T07:23:16.859 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:23:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:23:16.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:16.860 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:23:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:23:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:16.861 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:17.086 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:17.075+0000 7f8282c1ef40 -1 Falling back to public interface 2021-07-26T07:23:17.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:18.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:18.203 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:18.204 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:23:18.204 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:23:18.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:18.205 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:18.562 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:19.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:19.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:19.564 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:23:19.565 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:23:19.565 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:19.566 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:19.621 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:19.611+0000 7f8282c1ef40 -1 osd.0 0 log_to_monitors {default=true} 2021-07-26T07:23:19.911 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:20.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:20.913 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:20.915 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:23:20.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:23:20.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:20.917 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.0 up' 2021-07-26T07:23:21.298 INFO:tasks.workunit.client.0.smithi094.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/355287,v1:127.0.0.1:6803/355287] [v2:127.0.0.1:6804/355287,v1:127.0.0.1:6805/355287] exists,up 8c0e2c69-a90e-42fa-b48d-442025ee10fd 2021-07-26T07:23:21.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:23:21.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:23:21.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:23:21.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:23:21.304 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2021-07-26T07:23:21.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:23:21.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:23:21.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-07-26T07:23:21.305 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:23:21.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2021-07-26T07:23:21.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:23:21.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:23:21.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:23:21.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:23:21.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-07-26T07:23:21.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-07-26T07:23:21.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:23:21.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:23:21.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:23:21.308 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:23:21.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:23:21.308 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:21.309 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:21.309 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:21.309 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:21.309 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:21.310 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:21.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:21.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:23:21.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:23:21.310 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:23:21.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:23:21.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:23:21.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:23:21.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:23:21.311 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:21.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:23:21.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:23:21.312 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2021-07-26T07:23:21.313 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:23:21.313 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49d2c871-a547-495b-b203-cc954a27a2b9 2021-07-26T07:23:21.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 49d2c871-a547-495b-b203-cc954a27a2b9' 2021-07-26T07:23:21.314 INFO:tasks.workunit.client.0.smithi094.stdout:add osd1 49d2c871-a547-495b-b203-cc954a27a2b9 2021-07-26T07:23:21.314 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:23:21.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpYv5gPNkLExAAUrFSJkUeUoLyeVU+ttC1RQ== 2021-07-26T07:23:21.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpYv5gPNkLExAAUrFSJkUeUoLyeVU+ttC1RQ=="}' 2021-07-26T07:23:21.329 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49d2c871-a547-495b-b203-cc954a27a2b9 -i td/osd-recovery-stats/1/new.json 2021-07-26T07:23:21.829 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:23:21.844 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2021-07-26T07:23:21.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpYv5gPNkLExAAUrFSJkUeUoLyeVU+ttC1RQ== --osd-uuid 49d2c871-a547-495b-b203-cc954a27a2b9 2021-07-26T07:23:21.878 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:21.867+0000 7f6d0675ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:21.919 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:21.907+0000 7f6d0675ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:21.921 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:21.911+0000 7f6d0675ff40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:21.922 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:21.911+0000 7f6d0675ff40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:23:21.922 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:21.911+0000 7f6d0675ff40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:23:21.922 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:21.911+0000 7f6d0675ff40 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-07-26T07:23:22.026 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:22.015+0000 7f6d0675ff40 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-07-26T07:23:27.809 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-07-26T07:23:27.810 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:23:27.811 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2021-07-26T07:23:27.812 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd1 key to auth repository 2021-07-26T07:23:27.812 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:23:28.315 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.1 2021-07-26T07:23:28.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2021-07-26T07:23:28.316 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:28.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:23:28.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:23:28.317 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:23:28.344 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:28.335+0000 7fb800376f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:28.346 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:28.335+0000 7fb800376f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:28.349 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:28.339+0000 7fb800376f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:28.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2021-07-26T07:23:28.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:23:28.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=1 2021-07-26T07:23:28.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:23:28.655 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:23:28.656 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:28.656 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:23:28.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:23:28.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:28.657 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:23:28.902 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:28.891+0000 7fb800376f40 -1 Falling back to public interface 2021-07-26T07:23:29.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:30.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:30.035 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:30.035 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:23:30.036 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:23:30.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:30.037 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:23:30.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:31.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:31.380 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:31.380 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:23:31.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:23:31.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:31.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:23:31.457 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:31.447+0000 7fb800376f40 -1 osd.1 0 log_to_monitors {default=true} 2021-07-26T07:23:31.770 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:32.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:32.772 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:32.773 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:23:32.773 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:23:32.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:32.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.1 up' 2021-07-26T07:23:33.135 INFO:tasks.workunit.client.0.smithi094.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/356693,v1:127.0.0.1:6811/356693] [v2:127.0.0.1:6812/356693,v1:127.0.0.1:6813/356693] exists,up 49d2c871-a547-495b-b203-cc954a27a2b9 2021-07-26T07:23:33.139 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:23:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:23:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:23:33.140 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2021-07-26T07:23:33.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2021-07-26T07:23:33.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2021-07-26T07:23:33.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-07-26T07:23:33.141 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-07-26T07:23:33.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-07-26T07:23:33.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2021-07-26T07:23:33.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-07-26T07:23:33.142 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-07-26T07:23:33.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-07-26T07:23:33.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-07-26T07:23:33.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-07-26T07:23:33.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-07-26T07:23:33.144 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-op-queue=wpq' 2021-07-26T07:23:33.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --chdir=' 2021-07-26T07:23:33.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+= 2021-07-26T07:23:33.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-07-26T07:23:33.145 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: get_asok_path 2021-07-26T07:23:33.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-07-26T07:23:33.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-07-26T07:23:33.146 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-07-26T07:23:33.146 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-07-26T07:23:33.147 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.257957 2021-07-26T07:23:33.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:33.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' 2021-07-26T07:23:33.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-osd=20' 2021-07-26T07:23:33.147 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-ms=1' 2021-07-26T07:23:33.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --debug-monc=20' 2021-07-26T07:23:33.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-07-26T07:23:33.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-07-26T07:23:33.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-07-26T07:23:33.148 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-07-26T07:23:33.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:33.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2021-07-26T07:23:33.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2021-07-26T07:23:33.149 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2021-07-26T07:23:33.150 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2021-07-26T07:23:33.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5fddd2aa-7abe-4771-8b88-ad8898d68a9d 2021-07-26T07:23:33.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5fddd2aa-7abe-4771-8b88-ad8898d68a9d' 2021-07-26T07:23:33.151 INFO:tasks.workunit.client.0.smithi094.stdout:add osd2 5fddd2aa-7abe-4771-8b88-ad8898d68a9d 2021-07-26T07:23:33.152 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2021-07-26T07:23:33.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1Yv5gkxEcCRAAk2KNNzsEP2VHNIgp3BNhkQ== 2021-07-26T07:23:33.161 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1Yv5gkxEcCRAAk2KNNzsEP2VHNIgp3BNhkQ=="}' 2021-07-26T07:23:33.162 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5fddd2aa-7abe-4771-8b88-ad8898d68a9d -i td/osd-recovery-stats/2/new.json 2021-07-26T07:23:33.669 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:23:33.683 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2021-07-26T07:23:33.684 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1Yv5gkxEcCRAAk2KNNzsEP2VHNIgp3BNhkQ== --osd-uuid 5fddd2aa-7abe-4771-8b88-ad8898d68a9d 2021-07-26T07:23:33.714 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:33.703+0000 7f22ee4c0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:33.717 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:33.707+0000 7f22ee4c0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:33.718 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:33.707+0000 7f22ee4c0f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:33.719 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:33.707+0000 7f22ee4c0f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:23:33.720 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:33.707+0000 7f22ee4c0f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:23:33.720 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:33.707+0000 7f22ee4c0f40 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-07-26T07:23:33.794 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:33.783+0000 7f22ee4c0f40 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-07-26T07:23:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-07-26T07:23:39.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2021-07-26T07:23:39.823 INFO:tasks.workunit.client.0.smithi094.stdout:adding osd2 key to auth repository 2021-07-26T07:23:39.823 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2021-07-26T07:23:39.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-07-26T07:23:40.345 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2021-07-26T07:23:40.345 INFO:tasks.workunit.client.0.smithi094.stdout:start osd.2 2021-07-26T07:23:40.346 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=fa6e8135-887f-41e3-9780-27aa6d3964ca --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --osd-op-queue=wpq --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.257957/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-07-26T07:23:40.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2021-07-26T07:23:40.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2021-07-26T07:23:40.347 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2021-07-26T07:23:40.374 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:40.363+0000 7f8b7a271f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:40.376 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:40.367+0000 7f8b7a271f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:40.379 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:40.367+0000 7f8b7a271f40 -1 WARNING: all dangerous and experimental features are enabled. 2021-07-26T07:23:40.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2021-07-26T07:23:40.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: local state=up 2021-07-26T07:23:40.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: local id=2 2021-07-26T07:23:40.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: status=1 2021-07-26T07:23:40.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i=0 )) 2021-07-26T07:23:40.697 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:40.697 INFO:tasks.workunit.client.0.smithi094.stdout:0 2021-07-26T07:23:40.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 0 2021-07-26T07:23:40.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:40.698 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:23:40.925 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:40.915+0000 7f8b7a271f40 -1 Falling back to public interface 2021-07-26T07:23:41.071 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:42.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:42.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:42.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 1 2021-07-26T07:23:42.074 INFO:tasks.workunit.client.0.smithi094.stdout:1 2021-07-26T07:23:42.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:42.075 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:23:42.435 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:43.272 INFO:tasks.workunit.client.0.smithi094.stderr:2021-07-26T07:23:43.263+0000 7f8b7a271f40 -1 osd.2 0 log_to_monitors {default=true} 2021-07-26T07:23:43.437 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:43.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:43.438 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 2 2021-07-26T07:23:43.438 INFO:tasks.workunit.client.0.smithi094.stdout:2 2021-07-26T07:23:43.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:43.439 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:23:43.871 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: wait_for_osd: sleep 1 2021-07-26T07:23:44.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i++ )) 2021-07-26T07:23:44.874 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: (( i < 300 )) 2021-07-26T07:23:44.875 INFO:tasks.workunit.client.0.smithi094.stdout:3 2021-07-26T07:23:44.875 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: echo 3 2021-07-26T07:23:44.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: grep 'osd.2 up' 2021-07-26T07:23:44.876 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: ceph osd dump 2021-07-26T07:23:45.248 INFO:tasks.workunit.client.0.smithi094.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/358099,v1:127.0.0.1:6819/358099] [v2:127.0.0.1:6820/358099,v1:127.0.0.1:6821/358099] exists,up 5fddd2aa-7abe-4771-8b88-ad8898d68a9d 2021-07-26T07:23:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: status=0 2021-07-26T07:23:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: wait_for_osd: break 2021-07-26T07:23:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: wait_for_osd: return 0 2021-07-26T07:23:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2021-07-26T07:23:45.249 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-07-26T07:23:45.829 INFO:tasks.workunit.client.0.smithi094.stderr:pool 'test' already exists 2021-07-26T07:23:45.843 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-07-26T07:23:46.845 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2021-07-26T07:23:47.563 INFO:tasks.workunit.client.0.smithi094.stderr:set pool 1 size to 1 2021-07-26T07:23:47.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2021-07-26T07:23:47.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local cmd= 2021-07-26T07:23:47.576 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: local num_active_clean=-1 2021-07-26T07:23:47.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: local cur_active_clean 2021-07-26T07:23:47.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-07-26T07:23:47.578 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_timeout_delays 90 .1 2021-07-26T07:23:47.579 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: shopt -q -o xtrace 2021-07-26T07:23:47.579 INFO:tasks.workunit.client.0.smithi094.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: echo true 2021-07-26T07:23:47.579 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: get_timeout_delays: local trace=true 2021-07-26T07:23:47.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: true 2021-07-26T07:23:47.580 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: get_timeout_delays: shopt -u -o xtrace 2021-07-26T07:23:47.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: local -a delays 2021-07-26T07:23:47.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: wait_for_clean: local -i loop=0 2021-07-26T07:23:47.786 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: flush_pg_stats 2021-07-26T07:23:47.787 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: local timeout=300 2021-07-26T07:23:47.787 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ceph osd ls 2021-07-26T07:23:48.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: ids='0 2021-07-26T07:23:48.211 INFO:tasks.workunit.client.0.smithi094.stderr:1 2021-07-26T07:23:48.211 INFO:tasks.workunit.client.0.smithi094.stderr:2' 2021-07-26T07:23:48.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: seqs= 2021-07-26T07:23:48.212 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:23:48.212 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-07-26T07:23:48.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=21474836488 2021-07-26T07:23:48.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 21474836488 2021-07-26T07:23:48.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488' 2021-07-26T07:23:48.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:23:48.379 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-07-26T07:23:48.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=42949672966 2021-07-26T07:23:48.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 42949672966 2021-07-26T07:23:48.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2021-07-26T07:23:48.544 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for osd in $ids 2021-07-26T07:23:48.545 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-07-26T07:23:48.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seq=64424509444 2021-07-26T07:23:48.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: test -z 64424509444 2021-07-26T07:23:48.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444' 2021-07-26T07:23:48.692 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:23:48.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 0-21474836488 2021-07-26T07:23:48.692 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:23:48.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=0 2021-07-26T07:23:48.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 0-21474836488 2021-07-26T07:23:48.694 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:23:48.695 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=21474836488 2021-07-26T07:23:48.696 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-07-26T07:23:48.696 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.0 seq 21474836488 2021-07-26T07:23:48.696 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:23:49.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836486 -lt 21474836488 2021-07-26T07:23:49.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: flush_pg_stats: sleep 1 2021-07-26T07:23:50.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: flush_pg_stats: '[' 300 -eq 0 ']' 2021-07-26T07:23:50.054 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 0 2021-07-26T07:23:50.419 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 21474836488 -lt 21474836488 2021-07-26T07:23:50.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:23:50.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 1-42949672966 2021-07-26T07:23:50.420 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:23:50.422 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=1 2021-07-26T07:23:50.423 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 1-42949672966 2021-07-26T07:23:50.424 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:23:50.425 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=42949672966 2021-07-26T07:23:50.426 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-07-26T07:23:50.426 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.1 seq 42949672966 2021-07-26T07:23:50.427 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 1 2021-07-26T07:23:50.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 42949672966 -lt 42949672966 2021-07-26T07:23:50.819 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: for s in $seqs 2021-07-26T07:23:50.820 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: echo 2-64424509444 2021-07-26T07:23:50.821 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: cut -d - -f 1 2021-07-26T07:23:50.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: osd=2 2021-07-26T07:23:50.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: cut -d - -f 2 2021-07-26T07:23:50.822 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: echo 2-64424509444 2021-07-26T07:23:50.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: seq=64424509444 2021-07-26T07:23:50.824 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-07-26T07:23:50.824 INFO:tasks.workunit.client.0.smithi094.stdout:waiting osd.2 seq 64424509444 2021-07-26T07:23:50.825 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: ceph osd last-stat-seq 2 2021-07-26T07:23:51.145 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: flush_pg_stats: test 64424509445 -lt 64424509444 2021-07-26T07:23:51.146 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: get_num_pgs 2021-07-26T07:23:51.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:51.147 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:51.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: test 1 == 0 2021-07-26T07:23:51.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: true 2021-07-26T07:23:51.608 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_num_active_clean 2021-07-26T07:23:51.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: local expression 2021-07-26T07:23:51.609 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-07-26T07:23:51.610 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-07-26T07:23:51.611 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: get_num_active_clean: ceph --format json pg dump pgs 2021-07-26T07:23:51.612 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-07-26T07:23:52.000 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: cur_active_clean=1 2021-07-26T07:23:52.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_num_pgs 2021-07-26T07:23:52.001 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: jq .pgmap.num_pgs 2021-07-26T07:23:52.002 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_pgs: ceph --format json status 2021-07-26T07:23:52.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: test 1 = 1 2021-07-26T07:23:52.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: break 2021-07-26T07:23:52.466 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1615: wait_for_clean: return 0 2021-07-26T07:23:52.466 INFO:tasks.workunit.client.0.smithi094.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2021-07-26T07:23:52.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:52.467 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2021-07-26T07:23:52.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:52.597 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2021-07-26T07:23:52.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:52.700 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2021-07-26T07:23:52.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:52.774 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2021-07-26T07:23:52.849 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:52.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2021-07-26T07:23:52.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:52.916 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2021-07-26T07:23:53.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.016 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2021-07-26T07:23:53.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.101 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2021-07-26T07:23:53.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2021-07-26T07:23:53.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2021-07-26T07:23:53.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2021-07-26T07:23:53.401 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2021-07-26T07:23:53.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2021-07-26T07:23:53.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.577 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2021-07-26T07:23:53.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2021-07-26T07:23:53.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.727 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2021-07-26T07:23:53.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.802 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2021-07-26T07:23:53.884 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:53.885 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2021-07-26T07:23:54.031 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.032 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2021-07-26T07:23:54.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.178 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2021-07-26T07:23:54.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.253 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2021-07-26T07:23:54.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.328 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2021-07-26T07:23:54.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.420 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2021-07-26T07:23:54.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.495 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2021-07-26T07:23:54.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.570 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2021-07-26T07:23:54.645 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.646 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2021-07-26T07:23:54.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.721 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2021-07-26T07:23:54.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.821 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2021-07-26T07:23:54.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2021-07-26T07:23:54.966 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:54.967 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2021-07-26T07:23:55.033 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.034 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2021-07-26T07:23:55.108 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.109 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2021-07-26T07:23:55.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.392 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2021-07-26T07:23:55.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.468 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2021-07-26T07:23:55.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2021-07-26T07:23:55.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2021-07-26T07:23:55.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.693 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2021-07-26T07:23:55.767 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.768 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2021-07-26T07:23:55.841 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.842 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2021-07-26T07:23:55.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:55.968 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2021-07-26T07:23:56.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.068 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2021-07-26T07:23:56.176 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.177 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2021-07-26T07:23:56.251 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.252 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2021-07-26T07:23:56.326 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.327 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2021-07-26T07:23:56.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.402 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2021-07-26T07:23:56.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.477 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2021-07-26T07:23:56.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2021-07-26T07:23:56.652 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.653 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2021-07-26T07:23:56.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.728 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2021-07-26T07:23:56.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.803 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2021-07-26T07:23:56.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.878 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2021-07-26T07:23:56.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:56.978 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2021-07-26T07:23:57.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.054 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2021-07-26T07:23:57.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2021-07-26T07:23:57.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2021-07-26T07:23:57.301 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2021-07-26T07:23:57.376 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.377 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2021-07-26T07:23:57.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.459 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2021-07-26T07:23:57.534 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2021-07-26T07:23:57.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.609 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2021-07-26T07:23:57.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.710 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2021-07-26T07:23:57.784 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.785 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2021-07-26T07:23:57.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.860 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2021-07-26T07:23:57.944 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:57.945 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2021-07-26T07:23:58.052 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.053 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2021-07-26T07:23:58.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.165 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2021-07-26T07:23:58.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.231 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2021-07-26T07:23:58.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.307 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2021-07-26T07:23:58.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2021-07-26T07:23:58.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.449 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2021-07-26T07:23:58.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2021-07-26T07:23:58.607 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.608 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2021-07-26T07:23:58.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.674 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2021-07-26T07:23:58.740 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.741 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2021-07-26T07:23:58.837 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.838 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2021-07-26T07:23:58.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.914 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2021-07-26T07:23:58.989 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:58.990 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2021-07-26T07:23:59.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.064 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2021-07-26T07:23:59.150 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.151 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2021-07-26T07:23:59.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.257 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2021-07-26T07:23:59.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.332 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2021-07-26T07:23:59.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.399 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2021-07-26T07:23:59.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2021-07-26T07:23:59.540 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.541 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2021-07-26T07:23:59.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.632 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2021-07-26T07:23:59.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.699 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2021-07-26T07:23:59.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.766 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2021-07-26T07:23:59.832 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.833 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2021-07-26T07:23:59.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:23:59.975 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2021-07-26T07:24:00.073 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.074 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2021-07-26T07:24:00.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.143 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2021-07-26T07:24:00.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.211 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2021-07-26T07:24:00.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.284 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2021-07-26T07:24:00.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.385 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2021-07-26T07:24:00.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.460 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2021-07-26T07:24:00.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.535 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2021-07-26T07:24:00.610 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.611 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2021-07-26T07:24:00.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.685 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2021-07-26T07:24:00.789 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2021-07-26T07:24:00.856 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2021-07-26T07:24:00.931 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:00.932 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2021-07-26T07:24:01.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.007 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2021-07-26T07:24:01.082 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.083 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2021-07-26T07:24:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.175 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2021-07-26T07:24:01.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.248 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2021-07-26T07:24:01.314 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.315 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2021-07-26T07:24:01.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.383 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2021-07-26T07:24:01.473 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.474 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2021-07-26T07:24:01.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.549 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2021-07-26T07:24:01.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.616 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2021-07-26T07:24:01.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.691 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2021-07-26T07:24:01.758 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.759 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2021-07-26T07:24:01.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.850 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2021-07-26T07:24:01.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:01.942 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2021-07-26T07:24:02.050 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.051 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2021-07-26T07:24:02.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2021-07-26T07:24:02.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2021-07-26T07:24:02.302 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.303 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2021-07-26T07:24:02.378 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.379 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2021-07-26T07:24:02.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.453 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2021-07-26T07:24:02.528 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.529 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2021-07-26T07:24:02.628 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.629 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2021-07-26T07:24:02.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.703 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2021-07-26T07:24:02.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.779 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2021-07-26T07:24:02.853 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2021-07-26T07:24:02.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:02.929 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2021-07-26T07:24:03.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.021 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2021-07-26T07:24:03.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.096 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2021-07-26T07:24:03.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.171 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2021-07-26T07:24:03.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.246 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2021-07-26T07:24:03.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.321 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2021-07-26T07:24:03.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.408 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2021-07-26T07:24:03.475 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.476 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2021-07-26T07:24:03.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.543 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2021-07-26T07:24:03.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.618 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2021-07-26T07:24:03.719 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.720 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2021-07-26T07:24:03.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.798 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2021-07-26T07:24:03.891 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.892 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2021-07-26T07:24:03.983 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:03.984 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2021-07-26T07:24:04.088 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.089 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2021-07-26T07:24:04.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2021-07-26T07:24:04.266 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2021-07-26T07:24:04.341 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2021-07-26T07:24:04.411 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.412 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2021-07-26T07:24:04.487 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.488 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2021-07-26T07:24:04.578 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.579 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2021-07-26T07:24:04.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.654 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2021-07-26T07:24:04.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.730 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2021-07-26T07:24:04.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.805 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2021-07-26T07:24:04.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.880 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2021-07-26T07:24:04.980 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:04.981 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2021-07-26T07:24:05.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.056 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2021-07-26T07:24:05.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.131 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2021-07-26T07:24:05.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.206 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2021-07-26T07:24:05.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.306 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2021-07-26T07:24:05.381 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.382 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2021-07-26T07:24:05.464 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.465 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2021-07-26T07:24:05.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.548 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2021-07-26T07:24:05.623 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.624 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2021-07-26T07:24:05.723 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.724 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2021-07-26T07:24:05.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.790 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2021-07-26T07:24:05.857 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.858 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2021-07-26T07:24:05.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:05.941 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2021-07-26T07:24:06.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.025 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2021-07-26T07:24:06.126 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.127 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2021-07-26T07:24:06.200 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.201 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2021-07-26T07:24:06.267 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.268 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2021-07-26T07:24:06.342 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.343 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2021-07-26T07:24:06.443 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.444 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2021-07-26T07:24:06.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.519 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2021-07-26T07:24:06.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.594 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2021-07-26T07:24:06.667 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.668 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2021-07-26T07:24:06.742 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.743 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2021-07-26T07:24:06.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.854 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2021-07-26T07:24:06.936 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:06.937 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2021-07-26T07:24:07.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:07.020 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2021-07-26T07:24:07.103 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:07.104 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2021-07-26T07:24:07.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:07.187 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2021-07-26T07:24:07.290 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:07.291 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2021-07-26T07:24:07.364 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:07.365 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2021-07-26T07:24:07.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-07-26T07:24:07.440 INFO:tasks.workunit.client.0.smithi094.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2021-07-26T07:24:07.508 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-07-26T07:24:07.510 DEBUG:teuthology.task.console_log:Killing console logger for smithi094 2021-07-26T07:24:07.511 DEBUG:teuthology.exit:Finished running handlers